3 - Makemore 2 - MLP/3 - Makemore #2.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0fb6c136",
   "metadata": {},
   "source": [
    "# Building Makemore - Part 2: MLP\n",
    "\n",
    "[Video](https://www.youtube.com/watch?v=TCH_1BHY58I)<br>\n",
    "[Repository](https://github.com/karpathy/makemore)<br>\n",
    "[Eureka Labs Discord](https://discord.com/invite/3zy8kqD9Cp)\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "- [Deep Dive: Bengio et al. 2003](#deep-dive-bengio-et-al-2003)\n",
    "- [Building the Dataset](#building-the-dataset)\n",
    "- [Character Embeddings](#character-embeddings)\n",
    "- [Hidden Layer Construction](#hidden-layer-construction)\n",
    "- [Output Layer](#output-layer)\n",
    "- [Condensing the concepts](#condensing-the-concepts)\n",
    "- [Working with the full dataset](#working-with-the-full-dataset)\n",
    "    - [Accelerate with Batch-Processing](#accelerate-with-batch-processing)\n",
    "    - [Improve the Learning Rate](#improve-the-learning-rate)\n",
    "    - [Split the dataset](#split-the-dataset)\n",
    "    - [Remove the Embedding Bottleneck](#remove-the-embedding-bottleneck)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "977530a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\") # Use GPU if available"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "15d5d74f",
   "metadata": {},
   "source": [
    "In the last session we built Makemore with a Bigram model and then we built it again with a super simple single-layer neural network.<br>\n",
    "As already mentioned in the prior notebook, both of these approaches have serious limitations.<br>\n",
    "For example, all predictions are always based on the *single* previous character, not on larger context sequences of characters.<br>\n",
    "\n",
    "**And honestly: Neither of the two models really produce convincing names.**\n",
    "\n",
    "If we could use more than one character to predict the next character, we would:\n",
    "\n",
    "**a)** Have the two current approaches become really expensive really quickly<br>\n",
    "**b)** *BUT:* Be able to make much better predictions from more available context\n",
    "\n",
    "We had already noticed that the NN approach is noticably more flexible and scalable than Bigram.<br> \n",
    "We want to tweak it further. **We'll extend the single-layer NN to a deeper Multi-Layer Perceptron Model (MLP)**.<br>\n",
    "In order to do so, we will follow the general approach from [this very well written paper. \\[Bengio et al. 2003\\]](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2ce3db1c",
   "metadata": {},
   "source": [
    "## Deep Dive: Bengio et al. 2003\n",
    "\n",
    "With a vocabulary of $17,000$ words, the paper develops a *word-level* language model.<br>\n",
    "This differs from our *character-level* approach.<br>\n",
    "**We intend to keep our work character-level, but model our MLP just the same.**\n",
    "\n",
    "The paper assigns a $30$-dimensional feature vector to each input word.<br>\n",
    "A field $i$ in this vector does **not** correspond to the alphabetical index of the letter at position $i$ in the word. More on that later.<br>\n",
    "In fact, the paper inputs each word itself. A lookup table for the letter indices is responsible for the conversion. The lookup table is $17,000\\times 30$ in size. Each word (not character, but word) is an index in this table, which points to a unique $30$-dimensional vector.<br>\n",
    "\n",
    "![](./images/bengioetal03.PNG)<br>\n",
    "(This image is bottom-up, i.e. the input is at the bottom)\n",
    "\n",
    "> In the NN, **multiple words** are read in as input, i.e. multiple word vectors.\n",
    "> Example input: \"A dog was running in a [...]\"\n",
    "\n",
    "Let's say we have three words embedded into three $30$-dimensional vectors.<br>\n",
    "The resulting concatenation of these vectors forms the input for a fully-connected layer, activated by `tanh`, which can be chosen arbitrarily wide.<br>\n",
    "(It is logically decoupled from the amount of words in the input in its own width)\n",
    "\n",
    "Then follows another layer of the fixed width of $17,000$, where `softmax` is applied.<br>Softmax normalizes the outputs to partial probabilities over the set of possible words. In the end, the neuron with the highest assigned probability is chosen.<br>(And backprop is then performed, if training is running). This neuron represents an index for one of the $17,000$ trained words, the word to come next.\n",
    "\n",
    "The dotted lines in the image can be ignored. That is an idea that we will not consider further here.<br>\n",
    "So let's start (as with the prior makemore part) by creating the necessary data structures.\n",
    "\n",
    "To summarize the above, we need:\n",
    "- a lookup table for character embedding vectors\n",
    "- a fully-connected layer, tanh activated, arbitrary width\n",
    "- a fully-connected layer, softmax activated, width $17,000$\n",
    "- a loss function\n",
    "- a training loop"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5706505c",
   "metadata": {},
   "source": [
    "## Building the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e1a1fed0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['emma', 'olivia', 'ava', 'isabella', 'sophia', 'charlotte', 'mia', 'amelia']\n",
      "32033\n"
     ]
    }
   ],
   "source": [
    "# read in all 32033 words\n",
    "words = open('../names.txt', 'r').read().splitlines()\n",
    "print(words[:8]) # show a sample of the first 8 names\n",
    "print(len(words)) # this many words in total"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b62926ff",
   "metadata": {},
   "source": [
    "Just like in the prior lecture, we create two maps 'integer-to-character' (`itos`) and 'character-to-integer' (`stoi`) to work with the $27$ possible characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "82d06592",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n",
      "\n",
      "{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n"
     ]
    }
   ],
   "source": [
    "# build a vocabulary of characters map them to integers\n",
    "chars = sorted(list(set(''.join(words)))) # set(): Throwing out letter duplicates\n",
    "\n",
    "stoi = {s:i+1 for i,s in enumerate(chars)} # Make tupels of type (char, counter)\n",
    "stoi['.'] = 0 # Add this special symbol's entry explicitly\n",
    "\n",
    "itos = {i:s for s,i in stoi.items()} # Switch order of (char, counter) to (counter, char)\n",
    "\n",
    "# showing the two mappings, they really just are mirrors of one another\n",
    "print(itos)\n",
    "print()\n",
    "print(stoi)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9db318cd",
   "metadata": {},
   "source": [
    "To now build the dataset, we define a new paramter `block_size`, stating how many characters we will evaluate from the word in order to predict the next character. We set this to $3$ for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0ed72e77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "emma\n",
      "... ---> e\n",
      "..e ---> m\n",
      ".em ---> m\n",
      "emm ---> a\n",
      "mma ---> .\n",
      "\n",
      "olivia\n",
      "... ---> o\n",
      "..o ---> l\n",
      ".ol ---> i\n",
      "oli ---> v\n",
      "liv ---> i\n",
      "ivi ---> a\n",
      "via ---> .\n",
      "\n",
      "ava\n",
      "... ---> a\n",
      "..a ---> v\n",
      ".av ---> a\n",
      "ava ---> .\n",
      "\n",
      "isabella\n",
      "... ---> i\n",
      "..i ---> s\n",
      ".is ---> a\n",
      "isa ---> b\n",
      "sab ---> e\n",
      "abe ---> l\n",
      "bel ---> l\n",
      "ell ---> a\n",
      "lla ---> .\n",
      "\n",
      "sophia\n",
      "... ---> s\n",
      "..s ---> o\n",
      ".so ---> p\n",
      "sop ---> h\n",
      "oph ---> i\n",
      "phi ---> a\n",
      "hia ---> .\n"
     ]
    }
   ],
   "source": [
    "# Build the dataset\n",
    "block_size = 3  # context length: how many characters used to predict the next? (1 before)\n",
    "X, Y = [], []   # features (input) and labels (output)\n",
    "\n",
    "# Just showing off the first five words split into tokens\n",
    "for w in words[:5]:\n",
    "    print(f'\\n{w}')\n",
    "    context = [0] * block_size\n",
    "    for ch in w + '.':\n",
    "        ix = stoi[ch]\n",
    "        X.append(context)\n",
    "        Y.append(ix)\n",
    "        # Showing what input and expected output now look like\n",
    "        print(''.join(itos[i] for i in context), '--->', itos[ix])\n",
    "        context = context[1:] + [ix] # crop and append, like a sliding window; NEAT!\n",
    "        # old context starts at index 1 and is appended with the new index -> sliding window of block_size\n",
    "\n",
    "# Again, these *do not* carry characters, but their respective indexes\n",
    "X = torch.tensor(X)\n",
    "Y = torch.tensor(Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6578926d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  torch.Size([32, 3])\ttorch.int64\ttensor([0, 0, 0])\n",
      "Output: torch.Size([32])\ttorch.int64\t5\n"
     ]
    }
   ],
   "source": [
    "# This is what the dataset looks like right now\n",
    "print(f'Input:  {X.shape}\\t{X.dtype}\\t{X[0]}') # 32 times the 3 character indexes before the next one\n",
    "print(f'Output: {Y.shape}\\t{Y.dtype}\\t{Y[0]}') # 32 times the next character's index"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "adbc6db4",
   "metadata": {},
   "source": [
    "## Character Embeddings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42be37d5",
   "metadata": {},
   "source": [
    "For this dataset, we now can pretty straight-forwardly setup the NN structure from the paper. We start with the lookup table `C`.<br>\n",
    "We have $27$ possible characters ($26+1$).<br>\n",
    "<br>\n",
    "(The paper concerned $17,000$ words, which were compressed into $30$ dimensional representations)<br>\n",
    "<br>\n",
    "We will morph the $27$ characters into a numerical 2D space.<br>\n",
    "We will use $2$ dimensions per character for now, so we can visualize the results later.<br>\n",
    "This is not really compressing, but rather expanding the data. Still, works for our purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "754199ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.3851,  0.9870],\n",
      "        [ 1.1169, -0.8279],\n",
      "        [-1.8660, -0.0708],\n",
      "        [ 1.0652,  0.1053],\n",
      "        [ 0.0738,  2.1639],\n",
      "        [-1.1729, -0.1817],\n",
      "        [ 1.4497,  0.4343],\n",
      "        [ 0.7129,  0.7105],\n",
      "        [ 1.5698,  0.1488],\n",
      "        [ 1.1600,  0.7087],\n",
      "        [ 0.6667,  0.7811],\n",
      "        [ 0.5160, -0.2681],\n",
      "        [-0.1283,  0.8398],\n",
      "        [ 0.8666, -0.1507],\n",
      "        [ 0.6726, -0.0524],\n",
      "        [ 1.3076,  0.9398],\n",
      "        [ 1.6115, -2.2379],\n",
      "        [-1.6768,  0.4643],\n",
      "        [ 0.6039,  0.7399],\n",
      "        [ 0.4032, -0.5835],\n",
      "        [ 2.3729, -0.4736],\n",
      "        [ 0.7913,  1.3063],\n",
      "        [-0.4270,  1.7531],\n",
      "        [ 0.9208,  1.2955],\n",
      "        [ 1.4564,  1.8884],\n",
      "        [-0.4191,  0.0928],\n",
      "        [ 0.2173, -0.7426]])\n"
     ]
    }
   ],
   "source": [
    "# This will be the lookup table\n",
    "C = torch.randn((27, 2)) # each of 27 items has its own unique 2-dimensional numeric embedding in here\n",
    "print(C)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece5cc50",
   "metadata": {},
   "source": [
    "Before we embed all of $X$ into $C$, let's play around with this concept of embedding with a single numeric value first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "667e9d6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.1729, -0.1817])\n"
     ]
    }
   ],
   "source": [
    "print(C[stoi['e']])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5bf153e0",
   "metadata": {},
   "source": [
    "Ok, that was pretty straight-forward.<br>\n",
    "With our first iteration on Makemore, we used One-Hot Encoding.\n",
    "\n",
    "Let's try that out here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c81d6453",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Given 27 possible characters, embed the number 5 (e)\n",
    "F.one_hot(torch.tensor(5), num_classes=27) # shape is (27,)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "336841fc",
   "metadata": {},
   "source": [
    "If we now multiply this One-Hot vector with the embedding matrix `C`, we get the same result as before: `C[5]`.<br>\n",
    "**That is exactly what we want**.<br>\n",
    "\n",
    "Even though this construct looks like a detour, we now have capped the value of possible inputs (i.e. the numerical letter indices) at a known, desired maximum of $27$.<br>\n",
    "Remember, One-Hot Encoding allows for positional context and knowledge of range of value to make the whole thing much more robust.\n",
    "\n",
    "To drive the point home, this is what we now do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2503020c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.1729, -0.1817])\n",
      "tensor([True, True])\n"
     ]
    }
   ],
   "source": [
    "print(F.one_hot(torch.tensor(stoi['e']), num_classes=27).float() @ C) # stoi['e'] = 5\n",
    "print(F.one_hot(torch.tensor(stoi['e']), num_classes=27).float() @ C == C[stoi['e']])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cff6bdc2",
   "metadata": {},
   "source": [
    "Going further now, we will *not* do this, simply because a direct call like `C[5]` is sufficient for our example.<br>\n",
    "Still, the effect of One-Hot Encoding is very interesting in this case.<br>\n",
    "<br>\n",
    "**But how do we now get all $32\\times 3$ integer values from $X$ embedded?**<br>\n",
    "Where does the $32\\times 3$ come from? Well, we have $32$ inputs from the originally $5$ words we comprised our dataset of. And each of these inputs consists of $3$ characters. The $32$ thus is directly related to our arbitrary choice of dataset size.<br>\n",
    "\n",
    "Fortunately, Python is quite flexible when it comes to embedding.<br>\n",
    "For example, you can request multiple datasets like this: `a[[3,4,5]]`, or `a[[torch.tensor([3,4,5])]]` or even `a[[torch.tensor([3,4,4,4,5])]]`.\n",
    "\n",
    "We can use this to our advantage.\n",
    "\n",
    "As `X` carries the indices of the characters, we can use it to index into `C` directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4e699720",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1)\n",
      "tensor([ 1.1169, -0.8279])\n",
      "torch.Size([32, 3, 2])\n"
     ]
    }
   ],
   "source": [
    "print(X[13, 2])   # Plugging out an example character, here 'a' from ..a ---> v (14th input, 3rd character)\n",
    "print(C[X][13,2]) # This construct actually returns the 2-dim. embedding vector for 'a'\n",
    "\n",
    "# For all embeddings (character index to 2D vector) at once, Python can do this:\n",
    "emb = C[X] # Done!\n",
    "\n",
    "print(C[X].shape) # 32 times 3 times 2 -> 32 inputs, 3 characters each, 2 dimensional embedding vector each"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "561e264e",
   "metadata": {},
   "source": [
    "## Hidden Layer Construction"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c796ca18",
   "metadata": {},
   "source": [
    "It's time to build the hidden layer. We can make it as wide as we want. But one thing is important:<br>\n",
    "If we get three character vectors with two numerical values each, it makes for a total of $6$ inputs.<br>\n",
    "(Below, a layer width of $100$ is chosen arbitrarily)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c51dc916",
   "metadata": {},
   "outputs": [],
   "source": [
    "W1 = torch.randn((6, 100)) # 6 -> 3 vectors á 2 values, 100 neurons\n",
    "b1 = torch.randn(100)      # for each of the 100 neurons, add a bias"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fa61d7b9",
   "metadata": {},
   "source": [
    "What we essentially now want to do is $emb\\ @\\ W1 + b1$. But that doesn't work.<br>\n",
    "\n",
    "**Problem:**<br>\n",
    "The multiplication of $emb$ ($32\\times 3\\times 2$) with $W1 + b1$ ($6\\times 100$) doesn't work.<br>\n",
    "We'll have to find a way to reasonably attain a shape of ($32\\times 6$) for the multiplication.<br>\n",
    "\n",
    "**Solution:**<br>\n",
    "We concatenate the $32$ individual $3\\times 2$ vectors horizontally, instead of stacking them vertically on top of each other.<br>\n",
    "Thus, we obtain $32$ vectors, one per input, with $6$ values each, two for each embedded character, in order of their appearance.<br>\n",
    "There are many ways to achieve this. PyTorch is almost too versatile on this actually. Let's try the function `cat()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f72090b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 2])\n"
     ]
    }
   ],
   "source": [
    "# 32 vectors, one for each input tuple, each 6-dimensional (3 characters, 2 values each)\n",
    "emb[:, 0, :] # This is the first character's embedding vector for all 32 input tuples\n",
    "\n",
    "print(emb[:, 0, :].shape) # 32 times 2 -> 32 inputs, 2 values each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3970c307",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 6])\n"
     ]
    }
   ],
   "source": [
    "# Concatenating all three input character embedding vectors into one 6-dimensional vector per input tuple\n",
    "# 1 means: Concatenate along the first dimension (rows) -> 32 times 6\n",
    "torch.cat([emb[:,0,:], emb[:,1,:], emb[:,2,:]], 1)\n",
    "print(torch.cat([emb[:,0,:], emb[:,1,:], emb[:,2,:]], 1).shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "970aa0c9",
   "metadata": {},
   "source": [
    "This is exactly what we wanted.<br>\n",
    "To repeat, for the $32$ inputs, i.e. the $32$ triplets, we now have $32$ vectors of width $6$ instead of $3$ vectors of width $2$.\n",
    "\n",
    "**But, another problem:**<br>\n",
    "If we were to change the block size from $3$ to something else, we would have to adapt this concatenation code.\n",
    "\n",
    "**Solution:**<br>\n",
    "Instead of `cat()`, we can use `unbind()` to split the tensor into a list of tensors, which we can then concatenate again with `cat()`.<br>\n",
    "This way we can generalize the code to any block size, as the block size is now not hard-coded into the concatenation anymore.<br>\n",
    "We build in a generalized way, what we had above with `emb[:,0,:]`, `emb[:,1,:]` and `emb[:,2,:]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "890f0491",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 2])\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# We separate the 32x3x2 into a tuple of 3 individual 32x2 tensors using unbind()\n",
    "# 1 -> dimension 1 is the axis along a tuple of (therefore 3) tensors is created, each 32x2 (the other two dimensions)\n",
    "print(torch.unbind(emb, 1)[0].shape) # This is equivalient to above list [emb[:,0,:], emb[:,1,:], emb[:,2,:]]\n",
    "print(len(torch.unbind(emb, 1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "76aeec43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 6])\n"
     ]
    }
   ],
   "source": [
    "# We concatenate the three 32x2 tensors into one big tensor, just like before\n",
    "torch.cat(torch.unbind(emb, 1), 1) # concatenate all (3 here) individual 32x2 tensors along dimension 1 (a row is 3 2-dimensional vectors each -> 6-dimensional vector now)\n",
    "print(torch.cat(torch.unbind(emb, 1), 1).shape) # 32 times 6"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "97aa7a7a",
   "metadata": {},
   "source": [
    "Before we continue: **What have we done here?**\n",
    "\n",
    "We took the letters of our names. For each letter we recorded the $3$ preceding characters.<br>If there were none, this was filled and represented with the character `\".\"`, just like at the end of a word.<br>\n",
    "The respective follow-up letter is the output we expect given the input of the $3$ preceding characters.<br>\n",
    "The triplets are the inputs. But we embed them first, as we have seen above.\n",
    "\n",
    "For each triplet, we go through the characters and assign the alphabetical character index.<br>\n",
    "For each character index, we look up the corresponding $2$-dimensional embedding vector in lookup-table `C`.<br>\n",
    "**Every letter in the triplet is now represented by a $2$-dimensional vector.**\n",
    "\n",
    "The next layer has $6$ inputs and $100$ neurons. The $6$ result from the fact that we have a triplet as input, which is represented now by $3$ embedding vectors with $2$ elements each.<br><br>\n",
    "And at this point, the above problem arises: We have $32$ triplets total and per triplet $3$ vectors with $2$ elements each, but we need $6$ elements to represent a triplet for the next layer.<br>\n",
    "So we concatenate the $3$ vectors á $2$ values representing one triplet row-wise and in order into one vector of $6$ elements.<br>We do this for all $32$ triplets. Problem solved.\n",
    "\n",
    "**But not quite**, because we had hard-coded the block size of $3$ into the code.<br>\n",
    "This was generalized for any `block_size` through using `unbind()` with `cat()`.\n",
    "\n",
    "Actually, let's take a closer look at how tensors are mendable in PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "be905e88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([18])\n",
      "torch.Size([3, 3, 2])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.arange(18)\n",
    "print(a.shape) # 18-dimensional vector\n",
    "\n",
    "# Reshape to 3x3x2 tensor (works, as 3x3x2 = 18) (very efficient, no copying of data)\n",
    "a = a.view(3, 3, 2)\n",
    "print(a.shape)\n",
    "\n",
    "# Can we transfer this approach to our 32x3x2 tensor problem?\n",
    "True if torch.all(torch.eq(emb.view(32, 6), torch.cat(torch.unbind(emb, 1), 1))) else False\n",
    "# Long story short: Yes, we can! All of the above is equivalent to just re-arranging how we look at the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c32b1e1d",
   "metadata": {},
   "source": [
    "The `view()` function is very powerful.<br>\n",
    "It allows us to reshape a tensor in any way we want, as long as the total number of elements stays the same.<br>\n",
    "<br>\n",
    "More on this [here \\[ezyang’s blog\\]](http://blog.ezyang.com/2019/05/pytorch-internals/).\n",
    "\n",
    "Ok, now for real. We adjust the correct matrix `emb` with `view()`.<br>\n",
    "We do this right away with the full definition for the calculation of the hidden layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "be8387f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 100])\n",
      "tensor([-0.0766,  0.9645,  0.0392, -0.9835,  0.4891])\n"
     ]
    }
   ],
   "source": [
    "# h is the result of the hidden layer\n",
    "# h = emb.view(emb.shape[0],6) @ W1 + b1 # emb.shape[0] is 32, but this has to be variable (BUG!)\n",
    "\n",
    "# Alternative to the above comment, again because of PyTorch:\n",
    "# PyTorch reads the -1 and infers, because 6 is already used up from (2x3): 32 times 6 or inputs times 6 generally\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # produces 32 times 100 activations of the hidden layer\n",
    "\n",
    "# Just for fun, let's see what h and its contents look like\n",
    "print(h.shape)\n",
    "print(h[0][:5])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1da908c8",
   "metadata": {},
   "source": [
    "With the above, we glossed over the intricacy that is multiplying and adding tensors correctly.<br>\n",
    "It is the issue of broadcasting, again.\n",
    "\n",
    "For the above, we know:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8ecb53a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 100])\n",
      "torch.Size([100])\n"
     ]
    }
   ],
   "source": [
    "print((emb.view(-1,6) @ W1).shape)\n",
    "print(b1.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "67f87450",
   "metadata": {},
   "source": [
    "With Broadcasting we first:\n",
    "1) Align all dimensions to the right (of all tensors)\n",
    "\n",
    "This is the case here:\n",
    "- $[32, 100]$\n",
    "- $[(1), 100]$ (the $1$ is added by broadcasting)\n",
    "\n",
    "2) Now, expand the dimension of the tensor with the lower rank to match the dimension of the tensor with the higher rank\n",
    "\n",
    "This is the case here:\n",
    "- $[32, 100]$\n",
    "- $[(32\\times 1), 100]$ (we replicate the $100$ values of the $2^{\\text{nd}}$ tensor $32$ times)\n",
    "\n",
    "In effect, single one of the $32$ inputs is incremented by the same $100$ biases from the $2^{\\text{nd}}$ tensor all the time.<br>\n",
    "That's good, because we want to multiply each input with the same bias values.<br>\n",
    "Remember, we don't train/alter these yet, so they are strictly the same every time.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3f72d29d",
   "metadata": {},
   "source": [
    "## Output Layer\n",
    "\n",
    "This will look somewhat familiar. It's closely related to the setup of the hidden layer.<br>\n",
    "One thing is different though, apart from the different dimensions of the tensors: The activation function.<br>\n",
    "We use softmax here instead of tanh to get a normalized probability distribution over the $17,000$ possible words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1f6140f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits: torch.Size([32, 27])\n",
      "Probabilities: torch.Size([32, 27])\n"
     ]
    }
   ],
   "source": [
    "# Initialize weights and biases for the output layer randomly\n",
    "W2 = torch.randn((100, 27)) # 100 inputs, 27 output neurons\n",
    "b2 = torch.randn(27)        # 27 biases\n",
    "\n",
    "# Neuron outputs\n",
    "logits = h @ W2 + b2\n",
    "print('Logits:', logits.shape) # This should be 32 times 27\n",
    "\n",
    "# Softmax transforms outputs into probabilities\n",
    "counts = logits.exp()\n",
    "prob = counts / counts.sum(1, keepdims=True)\n",
    "print('Probabilities:', prob.shape) # This should be 32 times 27 still"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "85453f6f",
   "metadata": {},
   "source": [
    "Now we have to compare the output of the probability distribution of the last layer to the respective label found in `Y`.<br>\n",
    "To do this, first, we want to index into the rows of `prob` (the output of the last layer) with the indices found in `Y`.<br>\n",
    "And we want to do this for each of the $32$ input triplets respectively.<br>\n",
    "\n",
    "Let's quickly look at `Y` first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f637c7e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[0] # This is the first expected output, the index of the first character in the first word"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "01444399",
   "metadata": {},
   "source": [
    "This code now does what we set out above to do.<br>\n",
    "It's beautiful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "790a9693",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2.6577e-05, 2.1749e-05, 2.8386e-01, 1.9907e-06, 1.2809e-03, 5.1389e-06,\n",
      "        4.2525e-08, 1.3930e-02, 1.7646e-12, 1.4201e-05, 2.0729e-02, 1.0157e-03,\n",
      "        2.0616e-06, 8.6343e-08, 5.6801e-12, 6.5059e-06, 1.3552e-01, 6.4974e-09,\n",
      "        1.0175e-06, 4.9333e-16, 8.6279e-12, 3.6867e-03, 3.4089e-01, 2.1658e-07,\n",
      "        3.8372e-02, 1.7870e-06, 1.1894e-10, 3.3018e-01, 6.0720e-05, 7.3781e-12,\n",
      "        8.2838e-11, 3.5406e-03])\n",
      "torch.Size([32])\n"
     ]
    }
   ],
   "source": [
    "# For each of the 32 input triplets, give the probability for the correct output neuron, which itself is accessed at the index stored in Y at the same position\n",
    "print(prob[torch.arange(32), Y])\n",
    "print(prob[torch.arange(32), Y].shape) # 32 probabilities, one for each input triplet"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3810c280",
   "metadata": {},
   "source": [
    "These are the probabilities assigned to the indices for which we know from `Y` that they should be $1.0$.<br>\n",
    "With this knowledge, we now build the cost function to compare the predictions to the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "72b16ef0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.936897277832031\n"
     ]
    }
   ],
   "source": [
    "# We want the average log proability over all 32 inputs to be our loss\n",
    "loss = -prob[torch.arange(32), Y].log().mean()\n",
    "print(loss.item()) # This is to be minimized"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "52a56eb3",
   "metadata": {},
   "source": [
    "## Condensing the concepts\n",
    "\n",
    "The following is not really a recap, but a boiled down version of the above code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "647b831c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X: torch.Size([32, 3])  Y: torch.Size([32])\n",
      "3481 parameters\n"
     ]
    }
   ],
   "source": [
    "print('X:', X.shape, ' Y:', Y.shape)\n",
    "\n",
    "# Let's re-initialize the weights and biases\n",
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)        # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)      # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)        # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)     # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)         # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "print(sum(p.nelement() for p in parameters), 'parameters') # print the total number of parameters (nelements = number of elements) present in the tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "87fec854",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Allowing for gradient accumulation\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f54816c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2561509907245636\n"
     ]
    }
   ],
   "source": [
    "# 1000 training steps (epochs)\n",
    "for _ in range(1000):\n",
    "    # Forward-Pass\n",
    "    emb = C[X] # (32, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2 # (32, 27)\n",
    "\n",
    "    # counts = logits.exp()\n",
    "    # prob = counts / counts.sum(1, keepdims=True)\n",
    "    # loss = -prob[torch.arange(32), Y].log().mean()\n",
    "    \n",
    "    # Replacing the above lines with PyTorch's built-in cross entropy loss function:\n",
    "    # this function behaves better numerically and avoids pitfalls like infinities by subtracting the maximum value from all logits before exponentiating\n",
    "    loss = F.cross_entropy(logits, Y)\n",
    "\n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None # Reset gradients (done before each backward pass to avoid accumulation over multiple passes)\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    # Update\n",
    "    for p in parameters:\n",
    "        p.data += -0.1 * p.grad # Nudge parameter values in negative gradient direction (learning rate = 0.1)\n",
    "\n",
    "print(loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "352961a5",
   "metadata": {},
   "source": [
    "We programmed this implementation of Cross-Entropy with Softmax pretty literally so far.<br>\n",
    "This is pretty inefficient. PyTorch can provide significant performance boosts through internal condensing. \n",
    "\n",
    "This way:\n",
    "1. The forward pass becomes more efficient\n",
    "2. Backprop becomes more efficient\n",
    "3. The Cross-Entropy is \"numerically more well behaved\", i.e. it is calculated more robustly (see the code comment on this)\n",
    "\n",
    "With the above code, we train on only $32$ inputs. We overfit with our $3481$ network parameters, to be precise.\n",
    "\n",
    "> The bigger the parameter count and the smaller the input set, the higher the risk of overfitting."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "168de786",
   "metadata": {},
   "source": [
    "Comparing the logits (the actual network outputs) to the labels, we see that the predictions are mimicking the labels visibly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "78ab22c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.return_types.max(\n",
      "values=tensor([13.3437, 17.7879, 20.5832, 20.6042, 16.7390, 13.3437, 15.9747, 14.1889,\n",
      "        15.9158, 18.3894, 15.9409, 20.9284, 13.3437, 17.1212, 17.1498, 20.0637,\n",
      "        13.3437, 16.4564, 15.1328, 17.0537, 18.5905, 15.9655, 10.8739, 10.6874,\n",
      "        15.5062, 13.3437, 16.2394, 16.9563, 12.7426, 16.2141, 19.0840, 16.0213],\n",
      "       grad_fn=<MaxBackward0>),\n",
      "indices=tensor([ 9, 13, 13,  1,  0,  9, 12,  9, 22,  9,  1,  0,  9, 22,  1,  0,  9, 19,\n",
      "         1,  2,  5, 12, 12,  1,  0,  9, 15, 16,  8,  9,  1,  0]))\n",
      "\n",
      "tensor([ 5, 13, 13,  1,  0, 15, 12,  9, 22,  9,  1,  0,  1, 22,  1,  0,  9, 19,\n",
      "         1,  2,  5, 12, 12,  1,  0, 19, 15, 16,  8,  9,  1,  0])\n"
     ]
    }
   ],
   "source": [
    "print(logits.max(1)) # For each of the 32 inputs, give the index of the highest probability output neuron\n",
    "print(f'\\n{Y}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "362fb8fe",
   "metadata": {},
   "source": [
    "In some cases, though, the predictions are not equal to the labels. Why is that?<br>\n",
    "In pracice, we will actually never reach a $0.00$ Loss. This is because of triplets and labels like these:\n",
    "\n",
    "`... -> a`<br>\n",
    "`... -> e`\n",
    "\n",
    "**Given the same input, different outputs are expected.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dea69294",
   "metadata": {},
   "source": [
    "## Working with the full dataset\n",
    "\n",
    "Rewind. We start with the full dataset again. Forget the ominous $32$ inputs.<br>\n",
    "Now its this many:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "61ca130c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: torch.Size([228146, 3]) torch.int64 tensor([0, 0, 0])\n",
      "Output: torch.Size([228146]) torch.int64 tensor(5)\n"
     ]
    }
   ],
   "source": [
    "# Build the dataset\n",
    "block_size = 3  # context length: how many characters used to predict the next? (1 before)\n",
    "X, Y = [], []   # features (input) and labels (output)\n",
    "\n",
    "# Just show the first three words as example\n",
    "for w in words:\n",
    "    context = [0] * block_size\n",
    "    for ch in w + '.':\n",
    "        ix = stoi[ch]\n",
    "        \n",
    "        X.append(context)\n",
    "        Y.append(ix)\n",
    "        \n",
    "        # Showing what input and expected output now look like\n",
    "        context = context[1:] + [ix] # crop and append, like a rolling window; NEAT!\n",
    "\n",
    "# These DO NOT carry characters, but their respective index numbers\n",
    "X = torch.tensor(X)\n",
    "Y = torch.tensor(Y)\n",
    "\n",
    "print('Input:', X.shape, X.dtype, X[0])\n",
    "print('Output:', Y.shape, Y.dtype, Y[0]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "556fa8d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3481 parameters\n"
     ]
    }
   ],
   "source": [
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)          # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)        # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)          # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)       # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)           # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "print(sum(p.nelement() for p in parameters), 'parameters') # Network stays the same, so same number of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a77f3e99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Allowing for gradient accumulation\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b8bf023d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19.505226135253906\n",
      "17.084487915039062\n",
      "15.776531219482422\n",
      "14.833340644836426\n",
      "14.002603530883789\n",
      "13.253260612487793\n",
      "12.57991886138916\n",
      "11.983101844787598\n",
      "11.47049331665039\n",
      "11.05185604095459\n"
     ]
    }
   ],
   "source": [
    "for _ in range(10):\n",
    "    # Forward-Pass\n",
    "    emb = C[X] # (228146, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (228146, 100)\n",
    "    logits = h @ W2 + b2 # (228146, 27)\n",
    "\n",
    "    # Leaving this here for happy memories of low performance:\n",
    "    # counts = logits.exp()\n",
    "    # prob = counts / counts.sum(1, keepdims=True)\n",
    "    # loss = -prob[torch.arange(32), Y].log().mean()\n",
    "    \n",
    "    loss = F.cross_entropy(logits, Y)\n",
    "    \n",
    "    print(loss.item())\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    # Update\n",
    "    for p in parameters:\n",
    "        p.data += -0.1 * p.grad"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e85769b",
   "metadata": {},
   "source": [
    "### Accelerate with Batch-Processing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a4aa6e8b",
   "metadata": {},
   "source": [
    "It takes quite a while to do a Forward-Pass and Backward-Pass over so many parameters at once.<br>\n",
    "**The training process can be accelerated through Batch-Processing**.<br>\n",
    "We want to randomly draw some $n$ inputs from the dataset and train on this subset. Forward. Backward. Update. All done exclusively on this subset.<br>\n",
    "Then we draw another $n$ inputs and repeat the process.<br>\n",
    "We do this until we have gone through the whole dataset once. This is called an **epoch**.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a9a222be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([103513, 124548,  30102, 196316, 173561,  22807, 175138,  93001,  58736,\n",
      "        193246,  80398,  96386,  27045, 203271, 220983, 210524, 125308,  53568,\n",
      "        110357,  78431, 112748, 128428, 104928, 192159, 126222,  89012, 106840,\n",
      "         35414, 127155,  23364,  49568, 172827])\n"
     ]
    }
   ],
   "source": [
    "print(torch.randint(0, X.shape[0], (32,))) # Construct a tensor of 32 randomly drawn numbers between 0 and 228146"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ca30134c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3481 parameters\n"
     ]
    }
   ],
   "source": [
    "# Reset the parameters once again for this new approach\n",
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)          # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)        # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)          # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)       # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)           # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "print(sum(p.nelement() for p in parameters), 'parameters') # Network stays the same, so same number of parameters\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "aca5e993",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for current mini-batch: 2.8925647735595703\n"
     ]
    }
   ],
   "source": [
    "for _ in range(1000):\n",
    "    \n",
    "    # mini-batch construction -> 32 indices of 3-dimensional character index vectors within X\n",
    "    ix = torch.randint(0, X.shape[0], (32,))\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[X[ix]] # (32, 3, 2), a single batch, X[ix] grabs only the 32 indices, C then grabs the 2-dimensional vectors for each of the 3 characters for the current of the 32 triplets\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2 # (32, 27)\n",
    "\n",
    "    loss = F.cross_entropy(logits, Y[ix]) # Y[ix] grabs the 32 expected output indices for the current batch (very elegant)\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    # Update\n",
    "    for p in parameters:\n",
    "        p.data += -0.1 * p.grad\n",
    "\n",
    "print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e4a10d7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for entire dataset: 2.662804126739502\n"
     ]
    }
   ],
   "source": [
    "emb = C[X] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 100)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Y)\n",
    "print('Loss for entire dataset:', loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9ccbbdc7",
   "metadata": {},
   "source": [
    "This is much faster, but less accurate. Still, this is the more efficient way, because through the subset of batches we are optimizing with a more quickly derived, yet just slightly deviating gradient each time.<br>\n",
    "**We are thus staggering towards the optimum, but the direction is still roughly correct.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "de831d61",
   "metadata": {},
   "source": [
    "### Improve the Learning Rate"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "96d8c774",
   "metadata": {},
   "source": [
    "The batch approach reveals new questions.<br>\n",
    "If you're now just staggering around, what is a good learning rate?<br>\n",
    "So far it was $0.1$, but now?\n",
    "\n",
    "The following steps are recommended to find a good learning rate for the batch approach:\n",
    "1. Reset parameters\n",
    "2. Choose a small number of iterations, e.g. $100$ (for batch-processing)\n",
    "3. Through trial and error and resetting and running with new learning rates: Optimum lies somewhere between $1$ and $0.001$. (Otherwise the gradient is too strong)\n",
    "4. We build a set of possible learning rates with `torch.linspace`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4123b2c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lre = torch.linspace(-3, 0, 1000) # 1000 values between -3 and 0, linearly spaced\n",
    "lrs = 10 ** lre # 1000 values between 10^-3 and 10^0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "320f76aa",
   "metadata": {},
   "source": [
    "With the above defined learning rate candidates, we now, per candidate, draw a batch of inputs and train on the batch.<br>\n",
    "Then we plot the loss for each learning rate candidate. The learning rate with the lowest loss is deemed the best.<br>\n",
    "This makes setting the learning rate a much more educated guess. (Still somewhat a guess, though)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "babbde29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the parameters once again\n",
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)          # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)        # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)          # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)       # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)           # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True\n",
    "\n",
    "lri = []   # List of applied learning rates\n",
    "lossi = [] # List of resulting losses for each learning rate respectively\n",
    "\n",
    "for i in range(1000):\n",
    "    # mini-batch construction\n",
    "    ix = torch.randint(0, X.shape[0], (32,)) # (32,)\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[X[ix]] # (32, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2 # (32, 27)\n",
    "\n",
    "    loss = F.cross_entropy(logits, Y[ix])\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Set learning rate candidate\n",
    "    lr = lrs[i]\n",
    "    \n",
    "    for p in parameters:\n",
    "        p.data += -lr * p.grad\n",
    "    \n",
    "    # Tracking stats\n",
    "    lri.append(lr)\n",
    "    lossi.append(loss.item())\n",
    "    \n",
    "# print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "68301df3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjyklEQVR4nO3dd3wc5bU//s9sV1lJltUtueKCjbGxjQ2m2XRI4NJuSLg3l3T4AgmEJATCTXDKD5JwISSBkFySUG4gkAAhCRDANNtgDLaxjXHBTe6WVayyaqst8/tj5pmdmS3alVazu9rP+/XSCyOtVo/GsubsOec5jyTLsgwiIiIii9gyvQAiIiLKLww+iIiIyFIMPoiIiMhSDD6IiIjIUgw+iIiIyFIMPoiIiMhSDD6IiIjIUgw+iIiIyFKOTC/ALBwO4/Dhw/B6vZAkKdPLISIioiTIsgyfz4e6ujrYbIlzG1kXfBw+fBgNDQ2ZXgYRERENwYEDB1BfX5/wMVkXfHi9XgDK4ktKSjK8GiIiIkpGV1cXGhoatPt4IlkXfIhSS0lJCYMPIiKiHJNMywQbTomIiMhSDD6IiIjIUgw+iIiIyFIMPoiIiMhSDD6IiIjIUgw+iIiIyFIMPoiIiMhSDD6IiIjIUgw+iIiIyFIMPoiIiMhSDD6IiIjIUgw+iIiIyFJ5FXz8z6uf4CuPr830MoiIiPJa3gQfgVAYD761C69va8arW5oyvRwiIqK8lTfBRygsa3/++FBnBldCRESU3/Im+CAiIqLswOCDiIiILJWXwYcsD/4YIiIiGhl5GXwQERFR5uRl8CGDqQ8iIqJMycvgg4iIiDInL4MP9nwQERFlTl4GH0RERJQ5DD6IiIjIUgw+iIiIyFJ5GXyw5YOIiChz8ib4kKTIn9lwSkRElDl5E3wQERFRdmDwQURERJbKy+CDE06JiIgyJy+DDyIiIsocBh9ERERkqbwJPiTot7tkbh1ERET5Lm+CDz3GHkRERJmTn8EHB30QERFlTF4GH0RERJQ5DD6IiIjIUnkZfLDqQkRElDl5E3zoz3YhIiKizMmb4EOPiQ8iIqLMycvgg4iIiDKHwQcRERFZisEHERERWYrBBxEREVmKwQcRERFZKi+DD875ICIiypy8DD6C4XCml0BERJS38jL4eOK9fZleAhERUd7Ky+CDiIiIMofBBxEREVmKwQcRERFZisEHERERWYrBBxEREVkqb4IPzvYgIiLKDnkTfBAREVF2YPBBRERElmLwQURERJZi8EFERESWYvBBRERElkop+Ljnnntw8sknw+v1oqqqCpdddhk++eQTw2NkWcayZctQV1eHgoICLFmyBFu2bEnroomIiCh3pRR8rFixAjfeeCPWrFmD5cuXIxgM4vzzz0dPT4/2mJ///Oe4//778eCDD2Lt2rWoqanBeeedB5/Pl/bFExERUe5xpPLgV155xfD/jz76KKqqqrB+/XqceeaZkGUZDzzwAO68805cccUVAIDHH38c1dXVeOqpp3Ddddelb+VERESUk4bV89HZ2QkAKC8vBwA0NjaiqakJ559/vvYYt9uNs846C6tXr475HH6/H11dXYY3IiIiGr2GHHzIsoxbb70Vp59+Ok444QQAQFNTEwCgurra8Njq6mrtY2b33HMPSktLtbeGhoahLomIiIhywJCDj5tuugkfffQR/vznP0d9TJIkw//Lshz1PuGOO+5AZ2en9nbgwIGhLomIiIhyQEo9H8LXv/51/OMf/8DKlStRX1+vvb+mpgaAkgGpra3V3t/c3ByVDRHcbjfcbvdQlkFEREQ5KKXMhyzLuOmmm/D888/jzTffxKRJkwwfnzRpEmpqarB8+XLtfQMDA1ixYgUWL16cnhUTERFRTksp83HjjTfiqaeewt///nd4vV6tj6O0tBQFBQWQJAm33HIL7r77bkydOhVTp07F3XffjcLCQlxzzTUj8g0QERFRbkkp+Hj44YcBAEuWLDG8/9FHH8UXvvAFAMBtt92Gvr4+3HDDDWhvb8eiRYvw2muvwev1pmXBRERElNtSCj5kWR70MZIkYdmyZVi2bNlQ10RERESjGM92ISIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiS+VN8OFy5M23SkRElNV4RyYiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIkvlbfDh6w9keglERER5KW+Dj7v+viXTSyAiIspLeRt8rNzZkuklEBER5aW8DT6IiIgoMxh8EBERkaUYfBAREZGl8jb4kOVMr4CIiCg/5W3wQURERJmRt8GHJGV6BURERPkpb4MPIiIiygwGH0RERGQpBh9ERERkKQYfREREZCkGH0RERGSpvA0+WrsHMr0EIiKivJS3wQcRERFlBoMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyVF4HH81d/ZleAhERUd7J6+Djw/3tmV4CERFR3kk5+Fi5ciUuueQS1NXVQZIkvPDCC4aPf+ELX4AkSYa3U045JV3rTaujXf5ML4GIiCjvpBx89PT0YM6cOXjwwQfjPubCCy/EkSNHtLeXX355WIskIiKi0cOR6idcdNFFuOiiixI+xu12o6amZsiLsoosy5leAhERUd4ZkZ6Pt99+G1VVVZg2bRq++tWvorm5Oe5j/X4/urq6DG9EREQ0eqU9+Ljooovw5JNP4s0338R9992HtWvX4uyzz4bfH7u/4p577kFpaan21tDQkO4lERERURZJuewymKuvvlr78wknnIAFCxZgwoQJeOmll3DFFVdEPf6OO+7Arbfeqv1/V1eXZQEIiy5ERETWS3vwYVZbW4sJEyZg586dMT/udrvhdrtHehkxseWDiIjIeiM+56OtrQ0HDhxAbW3tSH8pIiIiygEpZz66u7uxa9cu7f8bGxuxceNGlJeXo7y8HMuWLcOVV16J2tpa7N27F9/73vdQUVGByy+/PK0LJyIiotyUcvCxbt06LF26VPt/0a9x7bXX4uGHH8bmzZvxxBNPoKOjA7W1tVi6dCmeeeYZeL3e9K06TVh1ISIisl7KwceSJUsSzsd49dVXh7UgK8myjHBYhs0mZXopREREeSOvz3b5yUvbcOEvVyIUZg6EiIjIKnkdfADAjqPd2Nnsy/QyiIiI8kbeBx9ERERkLQYfREREZCkGH+CwMSIiIisx+CAiIiJLMfggIiIiSzH4ACBxzAcREZFlGHyAPR9ERERWYvBBRERElmLwQURERJZi8EFERESWYvBBRERElmLwQURERJZi8EFERESWYvABbrUlIiKyEoMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDwDr9x3L9BKIiIjyBoMPAC9vbsr0EoiIiPIGgw8iIiKyFIMPIiIishSDDyIiIrIUgw8iIiKyFIMPIiIishSDDwAyeLgLERGRVRh8gAfLERERWYnBBwBJyvQKiIiI8geDDyIiIrIUgw8iIiKyFIMPIiIishSDDyIiIrIUgw9wtwsREZGVGHwQERGRpRh8EBERkaUYfBAREZGlGHwA+PhQZ6aXQERElDcYfADoGQhleglERER5g8EHERERWYrBBxEREVmKwQcRERFZKq+Cj1vOnZrpJRAREeW9vAo+qks8mV4CERFR3sur4IOIiIgyj8EHERERWYrBBxEREVkqr4IPKdMLICIiovwKPuRML4CIiIjyK/gIhRl+EBERZVpeBR/BUDjTSyAiIsp7eRV8JJrzseOoz8KVEBER5a+8Cj4umFUT92Pn/2KlhSshIiLKX3kVfNhs3O9CRESUaXkVfBAREVHmMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiohHxwoZDeOr9/ZleBmUhR6YXQEREo08oLOO2Zz9CIBzGp2bXorTQmeklURZh5oOIiNJuIBjGQCgMWQZ8/kCml0NZhsEHERGl3UAwcpBnfyCUwZVQNmLwQUREaecPRQKOvgGeKE5GDD6IiCjt9JmP3oFgBldC2YjBBxERpZ0++Ohj2YVMGHwQEVHaDYTY80HxMfggIqK0Y+aDEkk5+Fi5ciUuueQS1NXVQZIkvPDCC4aPy7KMZcuWoa6uDgUFBViyZAm2bNmSrvUSEVEOMAQfbDglk5SDj56eHsyZMwcPPvhgzI///Oc/x/33348HH3wQa9euRU1NDc477zz4fL5hLzYdbFKmV0BENPrpyy5sOCWzlCecXnTRRbjoootifkyWZTzwwAO48847ccUVVwAAHn/8cVRXV+Opp57CddddN7zVEhFRTuCcD0okrT0fjY2NaGpqwvnnn6+9z+1246yzzsLq1atjfo7f70dXV5fhjYiIcht7PiiRtAYfTU1NAIDq6mrD+6urq7WPmd1zzz0oLS3V3hoaGtK5pCi1pQUj+vxERGQsu7Dng8xGZLeLJBkbK2RZjnqfcMcdd6Czs1N7O3DgwEgsSfPoF0+O+7F/e/AdtPcMjOjXJyLKB8x8UCJpDT5qamoAICrL0dzcHJUNEdxuN0pKSgxvI2latTfuxzYd7MRDb+0a0a9PRJQPjLtd2HBKRmkNPiZNmoSamhosX75ce9/AwABWrFiBxYsXp/NLjZheRuhERMNmKLvw9yqZpLzbpbu7G7t2RbIDjY2N2LhxI8rLyzF+/HjccsstuPvuuzF16lRMnToVd999NwoLC3HNNdekdeEjRZYzvQIiotxnLLuw54OMUg4+1q1bh6VLl2r/f+uttwIArr32Wjz22GO47bbb0NfXhxtuuAHt7e1YtGgRXnvtNXi98csdREQ0uvj1W20HmPkgo5SDjyVLlkBOkB6QJAnLli3DsmXLhrOuDGLqg4houNhwSonk5dkus+riN7Wy7EJENHyccEqJ5GXwcflJ4zK9BCKiUc044ZQ9H2SUl8FHIsx8EBENH8sulAiDDyIiSjvjnA8GH1bY39aLi3+5Cs9/eDDTSxkUgw+Tf318JNNLICLKeeY5H4k2KlB6vLn9KLYe6cKz6xl8ZKWZCRpOu/rZGEVENFz6zAdg3HpLI6O1WzkepK07+48Jycvg44RxpZleAhHRqGYONnpZehlxrd1+AEBbjz/DKxlcXgYfREQ0svRlF4BNp1YQwcexngGEwtld5mLwQUREaTcQNAYbbDodeS0+JfgIy0BHb3aXXhh8EBFR2pl7PvqZ+Rhxrbpej7YeBh85Z8P+9kwvgYgop7HsMnwHjvXiy4+txZo9bYM+VpZltHRHej1au7O77yMvg48Cpz3hxy//zWqLVkJENDqZMx9sOE3dy5uP4I3tzfjTmn2DPtbnDxquebbveMnL4MNpz8tvm4jIMubggz0fqevsCwAAOnoDgz5W9HsIbcx8EBFRvhHBR4lHOTydPR+p86lzpzr6Bs9itJqDD/Z8EBFRvhE9H6WFTgDs+RiKrv7kMx+tpjKL+f+zDYMPIiJKOzFkrLRADT5YdkmZyHx0JhV8sOwyKmR7pzARUTYbMAcfzHykrEvt+fD5gwiEEo+nFz0flV43AJZdctatf9mU6SUQEeUkWZYjZRdmPobMpztrTAQi8YgXzDNqvACY+chZGznrg4hoSIJhGeIQ29ICFwBmPoZC9HwAQEeSwcf0ahF8MPORk7J7Kj5RbtjV3I3G1p5ML4Mspt9mm+tll6Nd/dh51JeRr63PfAzWdNqiBhvT1cyHzx/M6h1GDD6IaET4gyFc/tC7uOI37yI4SL2aRpdYwUd/jpZdPvfIGnzq1++g2ddv6dcNhWV0+yPBR+cg223FVtvJlcVw2CQAygFz2YrBBxGNCF9/ED5/EO29AW1YEo1uslprEc2RNgkoVud85GLmIxSW0djag4FgGBv3d1j6tbt1WQ8gceZDP1q9yuvG2GKl1JXNpRcGH3Fk+3HERNlOn/LtMv0ipdHnvtc+wek/ewstPr+2zdblsGnHWeTiePXOvoDWu7L1SJelX1vf7wEkDj70o9Urit0YW6TseGntyd6mUwYfceTiPxSibNIfiKTemfkY/V75uAmHOvqw8UCHttPFZY8EH7mY+dCXLbYezmzwkejfkCi5FLsdKHDZmfkgovylz3ww+Bj9RH9Cj+5VuMthR6FLCT6yufkxnvZeXfBhcebDZ8oWJvo3JGZ8VKhBR0WxOusji7fbMvggohHhDzL4yCci+OjWBR9uhw0ekfnIwWxyuy7zcbC9z9KfY3Pw0dEbP4shRqmLoGNskZr5YMMpEQ2Hvus9V+jLLoMNSKLcJsuyMfOhll2cdgkFrtwtu7SbbvjbLMx+mP/NJJrzIWZ8iOmmY9UgJJsndTP4SMJf1x3ALU9vGHS8LdFIeH3rUZxw16t48M2dmV5KSlh2yR+9AyGtMdNYdtH1fORg5uNYj/Hn1sq+D5/a8+FyKLfpRA2nIsjQMh/s+cht6/e1Y/WuVnzn2Y/wwsbDeG79wUwvifLQbc99BAD4n9d2ZHglqWHmI3/oM3Pd/lDs4COHMx/q2Iy09n30DgRx3f+tw1/XHYj5cbFDrH5MAYBkez7c6n9F2SV7Mx+OTC8gm1358GrD/w823pZoJOgHNuUSZj7yhz746PEHI1tt7TZ4XMpr3L5ACLIsQ5KkjKxxKETPx+z6Mmw60JHWzMeaPW14dctRfLi/A1fNr4+6LiLzMb68EHtaegbp+VCDD68SdIittsx8jBIyR39QBuRs8BHUz/lg8DGa6QdidQ9Eej5cDhsKXcprXFmGFpTkCpH5OP24sQCAnc2+tP17FNmKFp8f+9p6oz7e1adc04YxhQCUAD4cZ/6UGK1eGaPsImfpjYvBB1GWG8jRXiM/53zkDXPmQ7/V1uOI3GZybbutmPMxe1wpvB4HAiEZu5q70/LcrbqsxNq9x6I+7vMr/2YaypWyS1hWhonFfC5RdhENp2rmYyAUjvs5mcbgIwUyj5sjSlo/t9rmDf22UEPwYbfBYbfBZVduNdkyvPEP7zTiqodX40hnX8LHiSbP8iI3jq8tAZC+vo+2wYIP9ZpWFLu1vpnOGE2nsixHdruomY8Clx1F6i6jbC29MPggohHBCaf5oyeq4VQJMtxq1sPjjPR9CP5gKG1ZhFQ98d5erNvXjh/9c2vCxx1Tyy5jCp2YKYKPNPV96JtB1+1tj/q4aNL2epwoK1QO5+uIcbicT9djIxpOgch222wdNMbgIwVZWjojykp+/dkufdmZ+qX0MO52CRh6PgBEZn3oMh8/eXEbzr1/BV75+IiFK1UyBUc6lRNq//VxE1bsaIn5uGAorAXNY4pcmFknMh+daVmHPiOxp7VH6wERROajxOPQTgaOtd3WPFpdEH0frcx8EFE+MR4sF79ZLh+t2dOG0376Jt7YdjTTS0kLY89HyFB2AaA1nYqfiWAojH9sOgwA+PMHsbeajpT23oChafSuv39smMYr6A+VKyuIZD4+afKlZR3mAWDr9xlLL6JJW5/5iJVBjEw3dRner+14ydLttgw+UpCtXcNE2UhfdpETNMvlo+Vbj+JQRx9eHyXBh77no9s0ZAxAZMS6Gnx8uL9Du5G+s6vV0tJAk5r1KPE4UOV1Y29bL/53xZ6ox7WrWYbSAiccdhsmVRRp74/Ve5EqETTMG18GAPig0Vh6EXM+Sgp0mY+YwYdxxodQkeWDxhh8EGWxUA5nC/pNryY5aCxCpNi7/dnRgDlc+p6PgWAYPWp5RSu7OI0Np29sjwRdobCMlz9usmqpaOpSmkwbygtx56eOBwD876o9US8u23X9HgBQ5HZo48v3HesZ1hrCYRnH1IzERSfUAgDW6TIf/YFI9sjrcaKsQAkkOmPM+ogXfES22zLzkfOY+CCr9QxEfqm7HLn1z9W8rZJNpxEi+OgZJdkg89lD4sZt7vkQPxNvbmsGAMxtKAMA/FMtwVhB9HvUlnpwwawaAErmxvzzKbbZjimKlDMmjlVmbuyNMZcjFR19AYjXFReeoKxhy+Eu7edBn0kqdjsiDacJej7EgDFBlF1as/Rwudz6bUaUZ8wnW+YSfdkFYOZDr6VbZD5y9+9Xz/xzKiaDip4P/fku+9t6sbO5G3abhJ9eORuAstV0sG2v6XJUDT5qSj3wOO0oV4MLEZSYv4fywshNfcJYpfSyr3V4mQ+RrSgrdKKhvBDjygoQCsvYsL8DQGS6qdftgN0mobQwQdlFXacINgSt4dTHzEfOY+KDrObTTQYdCIZzqu+ImY/4tLJLDgeXet1+U9ag13gomr7n40215HLyxDGYUVOCkyeOgSwDL26yZteLCDJqSjyG/5qDH9HzMRKZDxF8jFWf++SJYwAAH6jzPkS/h9ejNOqKskuszEdbt3HAmFBbqgwnMwdV2YLBRwpy6Pc+jRLmm1MuTTsVswfEoVwMPhT+YEi7FvqyWi7rMfWuiKyBmPNRqJZd3t9zTNvlcs6MagDApXPqAAAvbY4dfOxq7sYvX9+ZtixRU5fIfCg357oyEXyYMh+mng9Al/loG17mQzSBilkcCyaWAwA27FeaTsWLjhK10TSy2yW6hCKeq6LIWHYRB9Id7ujLyt6xvA0+RK2RKJuZ09m5dM6LyHyIX7DDCT4SHao1XOGwjJc+OoI9LdYMvNLPXRi1PR89xp4PkV14ZUsTPlRLC+ccXwUAOHWKcm7K7jgDx36xfAd+8foOvPRRevpC9D0fgFJ+ASK7YITYPR9K8DHczEebaSKpmJ4qhq6JuTiRzEf8no+2HmMgI1SXeOC0SwiGZS3gyiZ5G3w8eM1JKX8Ox6uT1cwHsuVS8CEyH9Ulyi/FoR4u9/QH+zH3R8vxwoZDaVub3mOr9+LGpz7E7c9vHpHnN9MPk8rmnp4VO1pw6j1v4K3tzVEfe293G865722s3tUKIPJ9FLuVm6XYVi16Pm5Yehx+fNkJWDK9Ei6HDYunjMXkymIAQJUamPj8QcMQMkHsLEnXsCzR81Gtfl1RnjjcYbxBi4BX3/MxXi27tHb7h5WJiQQMynNPqVSCmiOd/ej2ByOZD48SdCTs+RAlHNOcD7tNQl2Z8r0dPDa8YGkk5G3wUa+eFEiUzcw3p1w6FVRkPsSr3vYhzkbYfEiZKPnRwfRMltQ7cKwX//PaJwCAQ+3WNDzqgw9/MIxgFpbSwmEZP3lxK4509uO1rdHbYF/86DB2t/TgRbVUIno+RKAp6Hs+Pn/KBDz2xYXY9qML8dRXT9Ee43U7tPHr5imfQCQjkY5Azdcf0AIjkfEQGRCxBVeIlfkoLXBqDarDKb1Eej6U61VW6NLmcjS29Gjfq8h8iDkfnb0BQ9+XPxiKnAFjajgFIqWXgxb9bKcib4OPoWDPB1nN/OoqlzIfIvgQryzbh7jlT8yGGIkSxff//rH2/FbtxjHfYM39Etng7R3N2KmWAGKl+kXpormrH8FQWNvZJG7oQqzt4XbRBKSSJEmbn9HsM2Yf/MGQlvEwN7UOxVG1/OD1OLQsjVhzdM+H2nBaaMwoTFCzH/uGUXpp7TZmPgBomaDdLd2G6aaAEpwASs+X/nwcESA5bBJKChxRX6e+TFnrgXZmPogoBT5z2SULXyXHI25ItWpDX9uQgw8l6Ej3ttSD7b14+5PIuR4+f9CSLIR5rLYvDTfVdPutbuJnouDjaJffEDyJUoYgyi6DqfIqn9dsCsz0fRiDZT5kWcYnTb6oAH3nUR8OdfSpz6c8f60uSNJ2hXT0G7IK4sZeXhRpOAX0fR9Dz3xoO1R0wccUXfDh0003BYAilx0ONWjT/3206YIYSTIGdQDQUM7Mx6jAxAdZLarsEsiN4EOWZW3C6Ti17nxsmJmPdAcfO44qZ3SIejsQ2eI4krIx89E7EMQ3/rwBX3l8HR56axc+aIxM24zVZ9Ckbks92tWvBU9uh03bEiokOxivSmQ+TI2R+j6MwYKPt3e04IIHVuKH/9yive9YzwAuffBdfOa37yEclrXttPogSQQifYGQ1ugZDIW17EPczEfr0LMJIhDXTyUVP4e7W7oNJ9oCSnYo1vkuLabyjZloLzjIzEeOY92FLBa91TbzN6pkDITC2j+XdJVd0h187DyqlBWOry3RUvBWbAc2Bx+ZHjQWCIVx01Mb8I9Nh/H6tqO491WlB2ZGjRdAdDmqbyCklSRau/3aNVNKGXbDY1MOPkzXRj97Y7CZKKL5dduRyJH3e9t60BcI4VBHH7Ye6dIyKfrMh8dp17bTHlH7PsShcpIU6bcQ0pP5iN6hMqVKyXzsaemJnOviiXztWCfbattsvfGCD2Y+RgUReviDIXzmd+/hPrVRjWikmOdA5ErDqX66qZij0N47MKSTbbXgI81ZiR1q8DG1yhtp6LMi+OjOnuBDlmXc/txmvLm9GR6nDV86bRIaygtQVujEdy+aASB6m7N+22ZYjvQ+FLsdKHIb+w7cyQYfJbHLLvo+jMF2S318SAk69Ne3uSvy53d2teKIacaHUKMrvQCRGR8lHuVQOb3h9nz0B0La37m+5+M4teyyp7VHm+chGk6BSN+HftaHVr4xzfgQRObjSGd/1jU253Xwcf1ZU4b0eS9vPoIPGo/h12/uSvOKiIx6TVsPcyX48KtNcZIUSXGH5dgp/MH0jVDPx65mpewyrbpYG+ZkZebDaVdq9Jmc9fHWJ8147sODsNskPPi5efjBJTOx8jtLsfEH5+MkdRZSz0DI0EdxpMP4KlrM5yiKEXy47MZMSDyi4dScFdJnPhKVXWRZxpbDym6o5i6/1rvRomtgfXdXa2S0uqk3pdbUdCoyO+Uxbuoi89HU1R9za/BgRM+Py26DV3e96soK4HbYMBAMY/sR5WezRJd1iTXrw7xl16zK64bLbkMoLGfdpNO8Dj6+e+F0rPvvc5N+vEgjB4Isv5A1zCPKc2W3i8h8eBx2OO02lKiv4IbS99EzAmWXcFjWdnNMrfaitMCasossy9oNtqFceVWayRHr4hj3q+bV49yZysRR0bjo9TghxZhOa76J7VaHsxW7IztIhGGXXXQ9H4n+/g+292mlCn8wrG2n1Qcza/cewz513kVtaezgQ/SyaNtsC40lF0CZNip+nvcPYX6GVioxNYnabRImVSiBjVi/PvMRa9ZHZMZH7LKLzSZhXJaWXvI6+JAkKeoY4kTCMXo+vvWXTVE3CKJ0MWc+ciX48KvNpmJ+g3gFOZTgo0+31TZdZ9sc7uxD70AITruECWMLLSu79AyEtK2Sk9RX0Jksu3yszlCZE2Pis90maT0H+lS/eVrm7hal98HriZH5SDr4UG7+Laattoc7jcFHvL9/8X0IotyiD2b6A2Ftgqh5S7AIPsTX0w6Vi5H5kCQJEyuG3vfR1hM/YBB9H4K+5yPW+S5a70icsgug7/vIrqbTvA4+UqX92Ot2ND334UH8ac2+TCyH8oC4UYlTQXMl+NAyH+q6I8FHaidsyrKsbbUNhuW0lZ1Es+nkimI47TbdEKeRPX5cvBIvctm1UkOmyi6yLGsD3GaPK435mFhHuR82lV326DIfRUNsOBXXoq1nwNCboC+7hMKyYcaF3pbDXYb/F9dZBB/mLb/msovo+RANqcfUn4Oywtg39eGc8dLqi18qEdtthRJ95kMru+h6PnrElt34L6JjNZ1mwwGVDD4QaSAaTFiWsaelG//3njHYiDWVjygdxKt+cRPIlZ6Pfi3zIYIP5ZfjsZ7UMgv+YBj6HtV0ZQnENtvjqpVf9pFmvpHNfIjfFZVet1aiyFTm42B7Hzr7AnDaJUyrKY75mFh9BuIGXadmC0RZrChW2SXJOR9ji1yw2yTIcmQAV99AKGrGSLy+j48PmzIfagZFXO+lMyq1j7kdNu3fk1Cn9XwoN+iOBD0fADBBLZk1DmG7bavIfMTYHqvf9g1EttoCkcZtfamnLcawMjPRdCoGjYXCMs65fwVu/ctGrWE1Exh8AHj5G2fg0yfWDvq4cFjG2fet0F4tEI00UXYRr3oGgrlR4hOlSLHbYewQMx/mslO6+iNEv8e0KmU7qVVlF33wUZTh4EOUKqbXeOF2xG4MLVWDMn2fgShNzB1fZnhs8TDKLjabpA3cEoGDCASKXHYtA2AeuieIzIeYKRPJfCjPdflJ47TH1pZ6ogZy6aecyrKs6/mIfVMXB8H9c9NhHEix7yOyPTZx5sNpl7SyJQAcp5ZkROlIluWYW3bNzJmP9xvbsKelB29sazYEN1Zj8AElYv/uhTMGfVwWZKooz4g0s3illisTTkXZxa1mPsT5GKlOOe01bTVO1416p5r5mKpmPqza7SJ6GvSZj0yVXQYruQCRzIf+uoimTPPJ4N4YmY9kt9oC+r4PJXDQTp8tK9BukrEyH81d/Wjx+WGTgDOnVWrPEQrLWhZlbsMYLcNtnsIKRGbR9KrZlnV7lSFrNaWxb+oXzKrG/Alj0O0P4lt/2ZTSkfWR7bHRzz1Zl/lQGn4jQZLoB2n2+dGlnlEjfh8k7vlQvm9xdtE/Nynn8Vx0Qk3SweFIYPChqowzpEVv1c7WmO9nTEIjIRSWtR4P0WyWKxNORebDY8p8pDpoLCrzkaYb9QH1F7HYXWBV5qOpK3KUerEnfZmPo139eG93G97a3oxAkgGqCD5m1SUIPgqNvTD6AWNzG8YYHhtrzkeyZRcgeseL6C2pLfVouz5iBR8i6zG5slgLMFp8frT3DiAUliFJys6S046r0J7PrMBl177XP77biL1tvRhT6MT5M2tirtVht+EXn5mLIpcdH+w9hv9duQc7j/pw32uf4L9f2JxwC26i7bGFLoeWvdH3eyj/79QO7tvV3I1W9TopB/PF39LcoGY+jnT2oW8ghH99rAQfl8ypi/s5VmDwofI47fjGOVMTPuYT9dUSkRX0zXVa2SVnMh/mno+hZj6Mv8TTkSUIhWWtaU/cACLBx8hmIbRek6ritJVd7n55Gxbd/QY+98gafPGxtXhu/cFBP0eWZa3skijzoTU5qkGZ2OlS6LJrZQCh2ONEoe4m6LBJsNmizxuJp6pEjFg3ZT5KPdquDxF8DATDeG93G/oDIe37OKGuxBDAiOcZW+SCw27D186YjLOmVeK/Fk+M+fVFE+rvVirn2nzxtElRwZTe+LGFuOvSWQCAn72yHef9YiV+/eYu/GnNfjz3Yfy/A5HZiVcqEdmPWCURUZbZ1dw96IwPoaLYDZfDhrAMPLv+ADp6A6goduOUyWMTft5IY/ChUzHIXyKRlUTJQZIiB0ylsttlw/52/PcLm7WTPK3UHxS7XYxbbdtT3E0yEmWXrr6A1sQqavpW7XbZro7+nlFboo0iH87ZLrIs41k12BA7okRPQCKHO/vR3huAwyZhujpGPRbzSG/Rh1Fb6sGYQqchs1HstsNmk7TSizOFrAcAVGqHyxl7PmpLC3RZImUdv1uxG597ZA3OvX8FXtqsvJKfVVdqGFYmnkfsBJlYUYTHv7QQ88YbMzaCyIgMBMMoctlx7akTB13zv8+vx0Un1Kjfr6RlXlbvjp0lB3SZjzilEhFgxDqlVgR8u5u7tfJNon4PQOmnqVezKSKw+tTsmqjTha3G4CMNsmHbEo0+/QPKDbzAadcaApPd7bJmTxsu/81q/GnNfryw4dCIrTEef5zMxzG1Bv/PTYdx5982D1oi6PWnv+witlF6PQ7tBmlF2aWzN6A1a06v8aLYrXzN4XxPu5q7caxnAB6nDTedfRyA5GapbD6oZAumVXsTpuzLTA2nYuhXbWkBJEnSshUAtO9HbLdNtZ8guuyi7qopiy677FK39x5s78P2JiWbNGtcie503H4tw1AVo8cjltqyyMj1/zx1gjbUKxFJkvDAZ+fi8S8txNo7z8X9n5kDAHhvd1vMowTC4Ugza7xS/8w6pZk1Vm+Kvum0NYkZH0J9uThgTgnoMl1yAYD4OaU8NNQY4pFVjZhRU4Ir59end0GU13oDyi/aAqdd+0WebPDxvb9t1v7cHuNI9JEW6fmILrvIsoyv/3kDAKVB8IJZsevqANAbSP9ul1gDpERjZc9ACIFQOOVX7cnY1hTZkVHicWo36eEEH2vU02fnTxij3axaEwQf/QGl5v/7VY0AEpdcAF3DqRqwibKLyBJUl3i0G5rITiilCv+wgw+xpbemtEALPsQUUxFYLJ4yFuv3tcNpt2H2uFIEQsov8fbegBa8VCXRzwcAter1czls+PLpk5Jet9thx1lqo+uJ9WUoctnR3hvA1iNdOMF0fTv7AlpzarydNP82tw7BkIwl0yujPibOf9nV0o0T68sADJ75ACI7XgBlW3G87I+VGHzonD61Ysif+62/bmLwQWklmtYKXHZt10CyZZcm3WTIZBsQ08kfp+ziD4axpzUymGmwg+b6TGWXdPR8tMXYRqk/Q6OrL5DUL/RUiZLL8bVKmSMdu13e39MGAFg4caxW+080u+Gmpzbg9W1HASjlvPPUkerxlJlGeuubQAFoDZBA5PsR/02l2RSIZAJEI+VhtexSV+qJZIlMwceNS4/D9BovgiEZXo8T4bAMp11CICRj65FOw/MO5vSpFXjgjZ34f2dN0TIoqXLabVg0eSze3N6M1btbo4IPEViVFjjjBmduhx3XLBof82Mi83HgWK/2d5FMu4A++Pj0nLqUenFGCssuOubpckSZpAUfhsyHMROwt7Unary/LBsnQcabCjlcv3l7F36/ak/Mj5kbTgtddu0Qrb9vPKw9brAdiuZ+CF8ago/2GDV3u03S1jdSpRdRHphRo6TVxU26dyCU0lZNQZZlvK9mPhZNLte2borZD7F8dLADAPDl0ydhxbeXaue5xGOecNqk2/4KwHCTFt9PkUv5byrbbIFIeaTF54evP6CVWJSttsY5H+Lk2kqvGxXFbm1OhzIvRLkOYhdMspmPk8aPwbYfXYhvnjctpXWbLZ6iNHK+s6st6mM71cMMxS6rVFV63fB6HAjLwLp9yt99UmWXMZFBmpecmPmSC8DgI63Y+0HpJIKGQlck+NBnPrY3dWHJ/7yNW57eaPg8fzBsKCH2D+HkzcF09Qfw81c+wU9e2mbIspjXLm5AkiRh4aRyAMDjq/dqj+sZSBxMmAOndGQ+RM/HGNMv7ZGe9bFNDT7EgCr9TorBrkMsja09aPEp5Y25DWWRzEePP+bvImXuhXLTvu7MyRifxGTnUnWLd1d/AGHdyajiZq8/I8VYdkm956NSDRoGQmG8ub0ZgNKXU+x26IaMBeEPRiafxgosxPtEOSiVLEY65l6ILb1rG49FZSo/0QLQ+E2+iUiSFGk6Vc/USSZLN6uuBDYJmF7txQnjSob0tdONwUcavb6tOdNLoFFEbDP16BpO9VttPz6kvLLbYdoCbt6eOhKZD/28BfEKTK9L3bKqL2eIsqb+5j7YkeRit4tDTROno+E03qFhI9l0GgrL+KRJ7HRRbjxuh037voYSVH2gZj3mNpTB47Rr308gJGu9EXptPX6EZcAmJXfDAiLXRJaV0os4SE3MjhBlF0mCts22eIgNpy6HTTtF9tt/3QQAOPd4JTOjn4kiGi2ddklbn565zJJs2SVdpld7UVHsQl8ghA372w0fE9mvRDuMBnOcKUM/2FZbQMnqv3DjaXjiywujprtmCoOPNNIfgkQ0XLEyH/ohY2ILbYfpZmkONkYi+OjV3SzXNkYHH+IGrg8+zojRUzXYK35RdhGp9O5hbEsVxPky5oa/kQw+9rX1oD8Qhtthw0T1UDJJkiI31SE00molFzWj5HHatdJHrL4Pbe5FsTvpbZYuhw1FLiWY2HigHb0DIRQ47ZhUodwAq9WsQrHLofURFA2x5wOIZCkCIRknTxyDuy+fDQDwusWcj0BkRH2xO+aNtNKU6Ui27JIuNpuEU6coP+vv7jaWXj5JQ/BhPvm2MslA8sT6spg7aDKFwUcabTviw6/f2BlVgycaCn3DqfhFPhDjxM+O3gFD46a5SXOw7MJQ9Oie84O97VEfFzfwMl3wMaWyOOo00cHWJj4utnR2xznbQ29Xsw8PvbUrbnOumDVSXmR81axN8xyB4EP/ild/4xf9EUPJ6IiMkyhnAZFXwbG222p9Eik204rttmLC88y6Eu17OK66GC67DZN1N8TiIZZdgMjhabPqSvCHL5yMAjXw0Xo+/EHD+TixZDrzAQCnqX0fq3dF5n30+IPaoXCi72coojMf1n9/6cDdLmn05w/2A1DGrQ82LZVoML1aw6kDbmd0z0dTp/JLOCwD3QNBbQpk34DxpjsSwbC+TLC9qQudfQFDCrxLvYHr3ydJEk6fWqENxVKeZ5Cyi7r2Ku34+cSPD4TC+Mrj67C3rRclBU58/pQJUY+Jd2hYZNBY+oMPMcrcXOsXN+qu/iD+sekwTpsyNqmbSSgsa1tJp1ZFnnNskQv72nq10oReS5eYe5Hazaq0wIlDHX14V72RnlAXuXFWeT1489tnGf6eh9rzAQDfPG8aplZ7cd2Zk7WfZyBSdvH1Dx586DMdRS57wimlI0X0fWw80IFufxDFboc2IbvS6457Wm4y9JNlbZIxwM8lzHyMgC2m452JhkJfdnHbo3e76CeX6m+Y5qmgI1F20QcfsgysN/V9iFKQeVDTFfPGwSZFUvJ9gcSv+EV5R6TSB8sQPL32APa2Ka8u39nZEvMxxzLQ8/H2J8pazCOtxU31vtc+wTf+vAG/fGNnUs/X1qMcnGaTjFstReDSFuP0YDHxM9UyhLguO44qg71mmbaP1o8pNIwCH+pWW0ApDXzv4uOjArDIePVASpmPZAeMpVtDeSEmjC1EMCxr2Y/hNpvqn1sEduVF7qzYNjsUaQ8+li1bBkmSDG81NfGHCI12x3oGRvywKhqdRPmkwGXXblLH1CFdQGTgExDZCglY1PNhKpdsPBAJuGVZ1n7mzQ2Bi6dU4KNlF+C2C6cDSCLzMWDMfMQ7Ul15bBC/0t2839vdFnMLa7yGU9GfEm8o24/+uRU3PvWhdhNJ1pHOPmw70gVJgjaMShCvyj9SJ4426magJHJUzXpVFLvh0N3kK7RZH9GZDzFjItUZFmWmAPKEBAfRAcBZ0ysxo8aLS+emb0unCGj6A2Gt3BivfKQPrlItMaXTEvXv+i018NT6PaqHF3zYbRImq1t1c/lIkBHJfMyaNQtHjhzR3jZv3jz4J40iEpRItG8ghHk/Xo45P3yN23ApZSJoKHDaMa1a6RVo7R5AU1c/AqGwtm0SMJ6ZIsos4he2uQyTDuZG0QNqLVv5WGRuRazdCPrTT81BjJlWdlFLBT0Dobj/ll7cdAQtPj8aygvgdTvQ1R/Eih3N+LeH3sW9r24HoJStxKwQc/DRoI6gbmyNPhulxefHH99txEsfHcFFv1ypPV8y3tqu3HxOaiiLekUvdobov04yRNbL3EA4Vpv1Eb/hNNWyiz74cNltmFqdeB7SlMpivHLLmfh0GudJFOtOeBVD6uKWXXTXpDLF7zWdlsyoAgC8/UkzZFnGdm230/C3uoqm02R2umSrEQk+HA4HampqtLfKyugxsfngsG73S7JDhEJheUQaBCn39OoaTj1qAAIAmw50osXnN8zy0O94EZ83Rm2oHErPx0AwjF3Nvrg3elF2Ean1g+2R4ENkPVx2m3bYmVmh2khoLhGZibKLeLUeCsvoD8QOpsQN+bQpFVikljdueXojNh3owBPv7YMsR06ztUkw9BQAkXT4jqPdWvZGnIGyVZ1O6rRLCMvAb1fsQTDJybFvblcmip6t3oz0ik39CLF6NWJ+r77YwYcIqGKNWBdll1SzAWLWB6BsEx6J0fODcep+lva0JA4+9NkAq3e66J06eSw8ThuOdPZje5MvbWUXAJiqBh+ZzOwM14j8FO3cuRN1dXWYNGkSPvvZz2LPnthTEAHA7/ejq6vL8JbrZCi/sPW/t5PNe1z8y1U4/gevoCuJrn4a3fp1PR8AcKJaa998qEMb9iToT2MVGZNy9VVwXyB+tiCeu1/ehnPvX4kVO2L3TYhyidgyeOBYJNAW/SclBc64MwUKXUlmPtSPjy12QTxVvL6PbjWQKXI7cNpxSvAh5l34+oM41jMQGTBW6IqqlU8cWwSnXUK3P4hDHX247dlNuOTBd7B6V6vWx3XezGo4bBJCYVkrYyTSHwjhXXXS5dIYwYe5GfKY2ssxmKOdIvgw3ny03S6xGk67h5/5mDVIyWUkiexHa3fing+3w66tORM7XQSP047F6pbbv6w7gPbeAGySsWF0qK4+uQFXnDQOX0rhDJpsk/bgY9GiRXjiiSfw6quv4pFHHkFTUxMWL16MtrboUbMAcM8996C0tFR7a2hoSPeSLBf5PS/HeF9ioiM61uwEyi/6IWMAcGKD8ov/o4OdhmZTwNTzoX5eufoLOBSWtQO3kiVepcU7nl1kLEQ25qivX2uGjfR7xN9lkGzmo09XQhpsW6rIxhS5Hdovfb3G1p7ITpcYuw1cDhsmq/MrPj7UpQVer29rxlZ1VPcJ40pRq24HPdQx+Fyf9/a0oS8QQk2JBzNjpNtF5mNSRREkSdm5pC+hxXNULaGYMx8VcRpOZVmOlF1S7fnQlc4yOR3T6zH+PFUWx/8+REZgqGe0pMtS9XC4pz84AACYWFGU8BThZNWWFuD+q+dqh8vlorQHHxdddBGuvPJKzJ49G+eeey5eeuklAMDjjz8e8/F33HEHOjs7tbcDBw6ke0mWC8synnhvLz7c12F4XyrYIkIiiIhkPsoAKMGHOfPREWNqqMh8AKk3nYoAwhdn+JUY9jW+vBAFTjtkOXIEemefcvOM1e8hRIKP2Ot68aPD+MHfP9ZuxAVOu5ZO1/eX6IlsTLHbjmnVxZhZW4KKYreW5m5s7UG7OmCsPM6JotPUxz7/4UGtvLNmT5sWfMyqK0VdqTLd83BHHwKhMO59dTseWbknqs/iWM8AfvTPrQCAc46vipkFOuf4atSVevCdC6Zra0qm70M0G5vnpoyN03Da1R/UDvtLNRugz3wM1mw6krymMlmi7+PMaZUocNoxf0JmT29dMl3Jdol/f+kouYwWI74BuqioCLNnz8bOnbG3kLndbrjduVu3iuX1bc1Ro9ZTDSYYe5C+4RRQShwuhw2dfQG8/Yny82VXSwD6zIdo0iwpcGgf7w+EEgYDZiL4iFf+69VKHHbUjynAzuZuHGzvxaSKosiAsTg3eCBx2SUQCuN7z282jAgvdNkxb/wY7G3rxdq9x3DmtOg+sm5d5kOSJLxw42kIhMK451/bsL3Jh8bWHu001nhzFmbUePHPTdBOfgWAbU2RUvDM2hKMG1MANCpnhyzfehQPvbUbAPDzV7fjx/92Aj67cDz6AyF85fG1aGztwbiyAtx8buy5P3MbyrD6jnMAAL98fSfaegYMjcTxiMyXuYQiGk6P9Q4gFJa1YWAtar+H1+NI+ZW36Plw2KRhTeYcLq+uROV1O7QBZLF8/9MzcduF07VjCTKlobwQx1UVaxnE6dXZca5KNhjxziG/349t27ahtrZ2pL9UVusLhPCjf27F6t2tgz8YPKQu13T1B9I+zEs/4RRQygLzxpcBiEybPFVtrBTZBv3nFbrsWuCSTBNztz+IJ97bixafX2vMjJf5EFmGIrdD2yUi+j7ibbPVS1R2+aDxWNTZJEVuB05WJ3l+oCtJ9gdCaFZvxKLsop+wWeR2aKPA97b1REarxwk+RBlJ33Yhy8pbldeNSq8b48oimQ9RnhLHuD//4SEASo3/w/0dKPE48PiXTk4q/V/hVZtFkwg+RL+JuewyptAJSVLWqy/fREouqb/Qm17jRYnHgSXTK9NSMhgqfdklmexNpgMPQZRegOGNVR9t0h58fPvb38aKFSvQ2NiI999/H1dddRW6urpw7bXXpvtL5ZTfrdiNP77biGseeT+pxzP0sM7Ww1145eMjQ/78/kAIS+59G0v/5+20Bo2RCaeRX6LmIVUXzVZm6MTq+Shw2rXAxVx2+fqfN+DKh1cbdmzc/txH+MHft+Bbf92kjU/vijOjRgQNhS4l8wFEdrykEnz0B8JRDZbLtx6NerzbYdPGiG840KH1l3z1iXVY/NM3cbijL9Lz4TImdCdVKMHRnpaeuKPVBfMMhjn1kTLDLHWypwg+DnX0aa9oz5mhHIAmdrjtVAdy/ecpE3BcVXI3HNGv0epL3PPhD4a03hVz2cVht2mTW/WlF63ZdAg9EOVFLnxw57l45L8WpPy56aQPPioy2EiaqqXTI43GLLtEpD34OHjwID73uc9h+vTpuOKKK+ByubBmzRpMmBA95jif7ElyeFCuCprmTuSSi3+1Ctf/6UN8dLBjSJ9/qKMPx3oGcKSzH3tae7Cr2Yf3TAdKDUVkwmnkl+6iSZHgY1ZdidYg2R5jt0uByxHJfOiCj+aufvxz02Gs39eOXS2RhtIXP1ICsJW6HS7xMx+RLEPDGDXz0W7MfJQkCD70uzz0a5NlOWbwIUnKYKWKYhcGgmFtC+ymAx0IhmXsbuk2lF30ROZjX1uvVq4wj1YX6scUaIFRkctu2E0gdnrU6TIfO5uVzIcoAzV19qujz/vU5xv82HpBCz4G+Xckshguuy1qABgQKSnpe1CGOuND8DjtGT8Ntdgd+V4zuYslVQsmlmNmbQlOGl+G8eXJ/zyMdmnv+Xj66afT/ZSjgvnY88HkWtXlmt+/jw8aj+GVW84Y1qFJVtMfyLa7pXtI3eP6ksb6ve247bmPAACrbluqlSSGwtxwCgAnjS+Dy2HDQDCM046riHkYmj5joi+7HOnsw+vbmlGiewUpXkEH4sysiNfz0TMQCYzMmQ+RhUmU+XA7bFp5oFc9+wJQ5mkc6uiDx2nD2TOq8PLmJu1zJEnCggnleGVLEz7YewwnjCvVyjNdfUFdw6nx11r9mAI4bBL6AiGtl+OEcbEbJ202CVOrvdh0oAPzJozRzugAlAPVACg9H1B6PsR1O+24sbBJQDAso7Xbr+2EEQelJUMEHy2DBR++SL9HrIBgbJELuwC06WZ9DHW0ejYxlF1yaL6Fy2HDS984PePBW7bh2S4m5x5fPSLPu68tdoe+nv5GmGs9H6IO/8za3NqtpL9pD7VGrD/n5F1dT08yWzHjCYdlLSOgr7N7nHYsnV4JSQIumFWjBR8dvQHtZ0Y/H8TjigQf33t+M77/wse4+emN2vOJnRXbjsSerxMv89HrjzScigBr19Fu/GvzkZgn2ppJkqSVR/RNp2+ojdqnH1eJZZfOQl2pB1fNr9c+vlDX96HfFdLZF9BttTX+PTrtNm2NgZCMadXF2jH0sZzUUAYAOHNqJSqK3ThjagVKC5za1xa7XXoHQgiEZBQ47WgYU6j1Xxzu6NP+7kVglgyxm2ewQWPiQEFzySXyPNFTTpsHOQ8lF6Ta85FNGHhE46m2JlfOG2fodLdSqttxs1G86ZOhsIxtR7ows7Ykqw5C0qe49Ye2DeaVj5swpbIIU6u9hlHjL2+O9I4M5VTPyFoi17HQ1NV/32fmoqmzH8dVFWvZkWBYRld/EKUFTq0fw+O0o8ApDnALaWdM6Ikb+Lq97THXES/zoS9x1JR44HU74PMH8f+e/FB7zGC7awpcdnT7g4brJ3bxnD2jClVeD969/WzDL+6T1IbbbUe6tFfzgBp8DBgbTvUmVRRp56b816kTE94Mbj1/GuZPGIMLT1D6aX5/7QKEwrJW/ipw2TG2yKVlFqZUFcFmk1Bb6sGRzn7sPNqtBW21pSkEH17R82HMfLy2pQmTK4u14VTxRqsL2nbbnlgNp5mdezEcuRx8UDRmPkzELxwrNXf1Y39br7HD3vJVpIc/zo6PH/1zCz7963fwi9d3WLyixPQp7u44r/LN1uxpw/V/Wo+Lf7UKHb0DhuyAfpiXP04glgz9LhDzDoNit0O7ERW47FqNX5Q9+tSva9jtEghFBTFAJPhYvy928OHrD0Zl4WRZ1rIVRS7lnJZXvnkmrpxXb3ic+URbsyKXcSdOR+8ANh7oAKAcTgZEv2IUGYxmnx8H2yOZpaNd/dq/n1hHqE8cqxzE5fU4cPlJ4xKuq8TjxCVz6rQx4m6H3dB3A0T6PgDguMpiw/vWqSf8lhU6UzrOvTJGz8e6vcfwtf9bj2v/+IHWmHvUF3ubrSC22/5uxR7MXvYqbn56Aw6oPxu5XXbJzZ4Pio3Bh0km0mML734DZ977llZ/B5RU/vKtR9O+fXOk9cfJHjz+3j4AwK/f3GXlcgxCYdlQ2gKMKW799k5ffyBuH8RLamNmICTj/uU74p7MGu9ayLKMn7y4Ff+3Zl/ctYqSi9th02Y1xNOgpvbFVlf9abhit0uLzx9zpoYIPjbFabYNheWozxsIhRFUr2OhWuIYV1aAOy6eAf1SB898KDdm0T/yzq5WhGXl3IpxZbEzBmOLXPA4bZDlyEmwALQGT0mKzhQBkWDma2dMTikgiEe/vqnqDhkt+FCzSPG+h3gi00kHtJ/Tt9Vs1aGOPqzaqfxZjFaPV3aZq2aHBkJh+PqD+PvGw1qgNtSG02yQqz0fFBuDjxg+t7Ah5iFQI22P7jTN7zz7Eb76xDos+8cWy9cxHPHKLpnW7OvH/J8sxy3PbDS8X5/iFqWEjt4BzF72Gj71q1VRzyPLsqEst3zrUUPPh168zMcHjcfw+3ca8f0XPo7b22Oe8ZFIvZoNiGQ+Ig2nImuib3ieUePVRn03+/wIhsLaxFRXjEPDzH0f+mCrUJeVqSh249Qpkd04gwUfhVrmQ3l+caNdMj16gJggSZJ2k/9wfyRbI9Zf5HLEfAFx1rRKfLTsfNx09nEJ15QsfeZjipr5EAPMxM62uhSDD1EuCYVlbWKtfi7QX9Yp/VTxRqsLZ02rxJo7zsHrt56Fp766CA3lkXVU5nDZRV9Oy+UMDikYfMRwzxUn4o9fOFn7/1g15HTRzzgIx7hXPZ1jDZzZmqlZs+cYOnoD+Memw1i3NzKkSp/i9qn9DWKb7I6j0eeabDviM4w27+wLxD1rJF4Pib5c0NUX+3O1bbZJDHUarw35UoMPXeBSoAUfyvcyp74Ur9xyJu64eAYAJfNx1KccZua0S5hRGz2HoKs/gJ1HfTj3/hX4+8ZDWrDlcdrgMAUr+pkGyQYfPX7l4DtxlsqS6YkDf5FR2HIo0iQrMh/mZlO9Ek/8g+5SNU7XSCpKYOb+jlQzH07d1tnWbj+6/UFs0mV3lm89irZu/6BlFwCoKfXguKpiLJ5SgRe/fgauWTQe1581JaUpt9lGlF1sEjCWmY+cx+AjCT++bFZan0+/wyKoizga23J/Fki2Bh/6zv/HVu/V/tw6SM+HuUwj5jqIYVS9AyHD36devGuh3wWjb5rU0w6VSyLzYZ6z0afb7SKCj51q5kP0TIiaeUu3H4fUz6stLYj5itLXH8DybUexq7kbz64/aOj3MLtyXj3GFrlwfG3JoNMwtSmngRD2tPagxeeH22HDgomJz+MQO0gGdGUx0VyZjpJKMkRg4bRLmDBWuabmbbWpBh+AftCYHx80tiEUljFhbCFmjyvVynxHOhKXXcxKC5y4+/LZuP2iGSmvJ5tMGFuIuQ1luGzuuEFLkZT9GHwksOq2pfjbDYtx2dxxWKxLJw/X5Q+9q6Xbg7oGxe+/8HFSn79iRwv+35/WRx1klQ2ytezSpMtW6P+s7/mIta3UPBl0v7plWky7BCKvumebZkdsO+LDM2v3RwUwe3SDvUQKPd7XjdW/YCbS6vuP9SIQCmtNr/oJp6JHQwQfYtfDsZ4B7FWD3nFlBdrNT6+rL6j1Gew/1qvtKimMkWUYU+TC299Zgr/dsHjQdWtbbf1BbFIbTWePKx10y3Oim/pIZin1ZtWVwG6TcNL4MVpjqrnMkmrZBYhst23p9mP1LiUDt3jKWHzmZOW07yff34++QAguhy2lnTSjgdNuwws3nob7r56b6aVQGnCrbQIN5YXaL+unvnoKvvvsR3hm3fDLIHtae/DCxkO4/KR6w2mkybr2jx8AUHZB/MKif4iBUBj9gVDUyZJm8ZosM01fKtGXSfQBnE99f0jXh9EzEDS8mt6nljYmVRRp20vFSO3Z9aXYfCiSJhcZlrAMfG7heO39jbppt/rMx+6WbnzpsbVo8fm1zERBEmUXkfk42N5raA4tcNmjsg+iRFNW4ITDJiEYlrXJruPGFGh9B3pd/QHt+h1q79NGrsfKfADRp4/GU6Cd7xLSgo856oyNRBJNDY23pnRrKC/E299eYjgjZmyRSxsABxhLM8mKTDkdwGq1/HfqlAosmV6Jx95tRFd/EGdPr8JVC+qT6gciylbMfKTgp1fOxpvfOistz/XNZzYpUyp/+uaQnyPe0eIj4fLfvIvZy14b9Ljv4WwvTYdwWMbfNx7SMhTCkc5IqUOf4WjXnYkiyi69uobKXtNOlv3qNR8/tlAbHy7S4MfXxp7sukI3X0OWZexpiQQf+szHa1uOYl+bEkCIoXQFSdxM68oKYJOUrJP4mbBJSvOoOXgRgYrNJmk3OrG9tc6U+RD9AV39Qe0Id2WUubL+4ZY4xOf3BULaGpIJPhLd1K0quwBKAKLPtEiSpDWdAqlNNxXE9d9yqBNb1cFvp04eixKPE298awnW3nkufnbViTh5YvwhaUS5gMFHCiRJwuTKYnz8wwvS8nxffOyDYX1+MGzdNJCP1ea+t7Y3J3xcpns+/rHpMG5+eiPOvPctw/v1mQ/9UCt9v4bPH4j6eI/p1FVxcx9fHgk+RL/BmEIn/u/LC1FXarzpiOcFlHS6T5d50Wc+9sY4/yeZhlN9Cl70pBSquz7Mr471Z0uIhkXxd1tvCj5EOcfXHzCUqj5Rj5hPpiSUiAiM2nsGtBvt3CTG2ycuu2Q2GyCCD5fdhoqi1JsiRcbr+Q3K6bjTq72caUGjEoOPIUhXXfndXcM7fCzWaaAf69L+IyFWWUXf05Dp4EO/NVEIh2VtKiSgZDhkWZn5oZ/gqWU+dKUL/Z/7AyEtAzC+vBClBcafgyK3A2dMrcSlc41DrPQ7WhpbjAFGsy7zEavhONnUuggUth9Rgg9RbvE4I//ETxpfZtx2aervGDfGGHyIQOVY94BhGNt29Rj54f47EDtT1u9vRyAko7zIZVhfPNUlHjjUhsOaEo+h+dDKzEcsos+jrswzpEm+YtsuoGTSvv/pmWlbG1E2YfCRBl/WnXxpJX3mY8dRH776xDp8+tfvxH98KIy/bTioNUgORazgQr8O0ezY4vPjxy9u1Y4cH66D7b14c/vgY+/ttugf6dYev2HyaDAswx8Mw+cPGg7wE9M89XM79H8+2N4HWVYmc5YXuaK2LYqbsds0Vt2nC3Cauoy7WwbLfCQbfIgJnmJSqQiMPLrmzZ9deaJhq+n4scbeCaXsEulhECWa3S3dhuskAhzz1M9UiZKSKEPNqS9Naius3SahRs0wVJe4DQflWdVwGo8492UozaYAcP7Mavzu8/Px4tdPx79uPgOnT60Y/JOIchCDjzT4zgXTM/J1Q+GwtmumMcaNy+yx1XvxzWc24cIHVqb0dfSDsPoDYTz89m78RTd/RJ+BCahbh+94fjP+8E4j/u3B6GCo2dc/aIbmre3NuOAXK7VmyIt/uQpfemwdfrdiNy598B187Yl1aPH5Icsy9rb2aGtw6F5tPrN2P656eDXe3q70XEwYWwhxb/P1B7XGSSEYltEfCBsCDn3m4xP1Ff+kyiJIkoQSU2OlaHY0N3nqJ6eKDIwoHYiejx5/UDv8S/89JNNwqnxvSvAhBm+JV9BnTa/EhbNq8Iur52BatXGGx3+eMsHw/7WlHowbUwCvx4FxZQVaut8870RscU00UyMZRabAKpl+D0Fcv0qvRyt/KWvKbPAxV/0e5qbwvejZbBIumFUT99RdotGCu13SYLB5BiNlx9FuzP3Rcjzw2blw2iM3rFBYjtoHPxAM4y310K6uJM8wEfSZjW1HuvCvj5Vjzv99QT0kSdICDkA5Il2WZa2BsCfGSO/Tf/oWBkJhLP/mmdpoarMvPrZW+e+ja7H+++dpa77nX9sBAB+hE2dOq4TX48DNT2/El0+fhO9/eib0L5y/+9xmAMA6NRtwyqSxONY9AJ8/qBxopgYZVV43Wrr9kGXg/AdWaGPKAWPmY606nGz+eGUORbzMh77UAQBdfcqJs5IkacHG7HGlONTRh2ZfvxJAqSWXMYVOjClyadmAZPsqJlUoWQrxVyUGXxW6HPjt5+fH/JwplcWYU1+qDbISP8evffNMuB12bZJrvNN5Fwyz6VH/vRW67Pj0iXVJf+64MQVAo5L50GePMh18nDuzGu98d6mWASGi2Jj5yAHxRnADSsPkFx9dayg39JqaJJ9bfxAzvv+vIfeY6M840e926RkIYfWuVlz3xHrT4+Wo0oOeeOW8pvGY4f2dfYGomRhtPQNxz0Dp7Avg/3tpGwDgD+80Aoh/BDygHMderKbou/uDkaPfC53wqjctfeABGDMf76vrPVk9Wj0q+PCIsosxYAiGZW2XjMh8zK5XXtn2B8Lo6g9ib6vy8YkVRYaj6JMNbEXmQxDBx2AevGYexpUV4IunTdTeV1tagPIil6GcAQATdWWa6hI3LhrmIYxlhZESz2//c37SawaUEeIOm4TFUyoMGahMN5wCylbgbDq5mSgbMfgYopMHmcKYTp/93zVaAJJMv4b5ILBv/XUThrMxJhCMfLL+uVd80oJrfv8+3ttjDGrEEKRU7G/rxck/eR03PqUcya4/YyTe8DVffxBjdDewB17fgfbegZiPBdTgQw0yfP6AFnyUFjgN8xr0xG6Xzr4Atqu7PBaqr/hLTMGHKEOYMx8AcNa9b+Nfm49oDabjywu1m3uLr1/LfEwaW2S4KSeb+Zhg6t/QNy4m0lBeiHe+uxR3XRI9xXd8uTGg0W/vvHJevTZca6hOnliOb503DU99ZRHOnBb/PJdY/m3uOGz50QX41Im1KNE1/mY680FEyWHwMUQNCQYdpdv7jcfwl3UH0NUfwOI4c0HEYCMgUip4fPVevLqlKeFzB0JhNLb2YN3eY/jxi1u1c0EMz63PfOh2PXz/77GDgv5AyJD5iJe50a/5uQ8PYiAU1ko6yWwv9PUHDH0HD7y+UzuczGxGjVeZy6DLfHT0RoIP/Q1fT8z5+HBfO2RZefVfpY611mc+nHZJy3jEy1a8uPmIdi5HdYlHOxjsaJcfWw4rpY8pVcWGzEeyPR+FLgeqdWd9TEkhixCvyXNmXQlOUk9IBYDpNV7MqPGiotiFL542/CZru03C18+ZisXHDa2pUlzv0izq+SCi5PBf6hDdcfHx8PmDuEY3uXIkffe5zQl/sepv5L0DIazc0YK7EpyI29kbwIqdLXj+w4OGG3ZYlqNeBccruxzriZ1l6DdlPn6q9mmYHeuJPJf+BvLxoc6ktuz6+oNx12B2unqDE5mPbn+k7FJS4Iw7M0VkPkTJZeGkyKt//Zr1syfilZwGgmGt7FJd4kZViRs7m7txtKtfO4Z9/oQxhvNmUpliOWFsEY52+VFb6knbro+vnD5Zy0ZVFLvxwo2nwR8MZ9UBZcayC3+lEeUCZj6GqNLrxiP/tQBLZyQ+gTOdbnpqQ9yPDYSMsyn+64+JB5h9/ekN+MafN0RlCl7bEtnO+kmTD9/722atXyFZfabMx+9W7on5OH3goM+ufPrX72iDuxLx9Qfino2iV1PiwdfOmgwA8KqZj/beADYeENtSnSgfJPMhmk31pQevrifihiWRo9r1mQ+P04Y5an/H9qYu7eybKq8H1er5Kh/ub0ezzw+nXcKc+jKUFUTWkmzmA4j0ZKTSOzGYC2ZVa3+eVaccFpdNgQdgLH9ZNV6diIaH/1JHiUdWNmp/7kjQ9yCs3BG7PHGoow+/fH0nbj53Ki759TsYCIXx9iBTTc36BkJJzWto7R7Aj/65FRVeV8JG0XgOd/RrB7AtmDBG29WiV+Jx4L07ztbWI25OP35xq/aY0gInJMReb89AEP2BkLblV5/5mFrtxZhCJ6ZWeXHl/Hrt/YbBXg1jcNelM3HhA6u0ZtYSjwMFLjsq1TLJKx8rAd+sulIUuOzasepAarM05jSU4S/rDmLe+PT1IznsNrx3x9k42N4Xd2dSpumDD2Y+iHID/6WOkHnjy/Dh/g7Lvp4YTw0g5UyF2S9e34Gbz52qZSMOd/YP8hlG/YFwUmWT17cd1YZXnT+zOvGDYxCjxMsKnfjJ5SfgwgdWRT2mtNBpCISKPdE/8sVuB+xxgqXegRDW7j2GQEhGlddtGE9eWuDE+987F5IEw9Zm/W6XIrc9ahy46PUQmQ9RZlkwYYz2/QgFruSTk589eTxm1pZgZl3sM2aGqra0IKtPUNXvyhnu7BEisgbLLiPk1vMyM3gMAN7bPbyx7cDwxqT3B0IxG1fN9H2or20dfHrp8zcsxtLplbhynpJlEG0aNSUebcKnmXkQmDfGK+MClx1l8Xa7+IN4RW2CPXtGVVRGx+WwRe360Gc+Cl0OeD1OQ4lGBB9VJcam2gUTo+eHFDiTf30gjngf7Ej60YYNp0S5h8FHmogboqB/hf3wf8yzdC1vpFgm0RNbXK/7v/WDPDK+/cd6sTPOWPW/XHcqxhTG7xl44Oq5Md//3P9bjHnjx+DRLy7EDUunGD52Yn0pPE47bj5natTnmYOPiRWRIGVadTGunFePy+aOi7smX38Qr6p9MBfNro27bj1z5gMwNqSK6ZciCBHmT1BKOkPZapvPRNnFYZMSzpchouzBlwlpcu9VJ6KqxI2H394NAIY0fnWpB+/dcTZOvSd6m+yK7yzB7pZufOmxdSl9PZfdZmjSTAeP04YJ5UX45KgPK+L0hJjVlXqiyjKJdtksnFSOtXeeixnffyXmDpOyQifOmFqBVTuNB8TpgwOvqXSycNJYAMA3z5uG06dWwOtxaCWYYNh4jS4/aRwqit3Y09KNK+fXw6sGJ/EaTkU5q8TjwKmTx8b9vvTcpsyH+L6EC9XhXFW67cQTxhZq24sNmQ8GH4OqLfVAkpRgLpleIyLKPL5MSBObTTLMZ9D3ANgkCbWlBfjgznOiPm/C2CIMBFObAPat86Zh9R1nD32xcZR4nFGHjQ1mKE2IDrvNcH1m6XoUygpd+NVnT8Klc+pww5JIhkM/TMyczVikawI9eWI5ZtREnq/TdH6LJEk4c1olvnDaJC3wEF9Xr36MscfhxPqypAen6Xe7iJLMEV2AJr7fKm8k8zF/QqRJVN806RrmIK98UFtagD9eezJ+F2eMPBFlH/5mGyEO3VkrIgtiPsJcSFSGiOXyecqr96e/dkrCx12zKHoGyaSK2L0RgPLKcUZNasHE4inJZQMA4MeXnaD92a+bS/LVMyZrfxbTRn/1uZNw6dzIWR/6HQ361LrHaYsKFPTMwUc8Y4oiz7/8m2filVvONHw80XUz069PxA63njcNAPCNs4/TXp0XuOxaFmfBhEgAVVHswqdOrMXFs2sMGROKb+mMKh7GRpRDGHyk0TnHKzs2qkvchlf2ogFRkiR86sTovoGFk8rxjbOPi3o/oPRImIkbsTkDcGK98Zfv8bXRux4S3US/csYkTE8i+LhkTiQomJZCsPJ53Smq4jrMqS/FxbNrMaWyCLWlHtSWRrIB06u9+NJpk/DdC2cYrqc+tX7BrJqYqXYx7XPRpOSCI0NmpcCJYrfDENRMrkw++NBnK8SZO5fOqcOK7yzBN9UgRJjboGRUzpwWmfIpSRIeumYefvMf81lGIKJRiT0faXRcVTFW3342yotc2iRLwNiA+NA181BZvAWPrd6rvU+SJNx6/nR86fRJeHb9QXz6xDrc++onOGFcScy5BcVqH8HMuhJ8+fRJ2qFq9141B79YvgOvbInszvhofj3+uv6g9rnmJkdAGQ3+0jfOwLRqL7Ye7or6uNmnZtei0GnHgfbeqMzHwknlGFdWgL9tOJTwOX56xWzMqivBZxY0wOWw4cWvnwGbzXitJEnCDy6ZGfPzL51ThzV72vC9i4+P+fG/XrcYz354EF9YPHHQ7wdQSiX//anj4Q+GtWs0o8aLg+3KbI7JSZ6VItYtiKyXJElRh78BwO+vXQBffxAVcbJiRESjEYOPNKtTdzUY5j6YDhpz2mO/mi0rdOEragnivs/MAQDsPOozPOYb50w1nJh5zaLxWvDh9Thw2tQKLfgodjlw77/PwTfPm4anP9iPkyeVxzzZ1utxYprau6Hf/lld4o6aIPr7/1qAc46v0pomzf5y3alo7xkYNPjwepyGqaCpNlb+6nMnIRgKwxGnJ2L82EKt1JGsr+jKP4ByPsrr25SdQ5NTKLvoDday4XbY4S5mUykR5RcGHxbwmOYufO3MKXh1y1FcNb8+zmdE6G+uP/j0THzpdOOBXjbdq2yvxzgsy6MOqKorK8Ct5ytzR8Qr+QKnXZsOqg+U9Ls+Yk39PDeJYWBjilxY+Z2l8DhtWHj3G4M+fqjiBR7poi9r1ZUNbciWKLsQEVEEg48REgxFdrCYMx+VXjdW3rY0qefRZ0liZUwqiiPBQpHLAV0cEXOnxOUnjUOx24FTp4zFgp+8DsC4LVifVRmOVHfNZKMr5o3DL9/YiUWTyg0BWir0p8ISEZGCwccI0TcwDme7pH56pnmSJqCUL1695Uy4HDbYbJIheIjVrOhx2g0NowDi3lhlGLcA67e05oPa0gKsueOcIZ0X8ua3zsKelh6ckuRsECKifMLgY4SUFjrx1+tPhVsNCoZKH7jEOfXdsENlSgq7MgRzZeD42hJsO9KFf5/fgAff2gVAaRCNN+HT5bBhIJjegWfZojzO2PXBTK4sTqlJlYgonzD4GEH649eHSj/nYXdL7JHlevMnlONnV87GpIrBb3xXzqvHcx8exC3nGBszn/7qKdh8qBMN5QVa8HHZSeMMw7P0Cl32URt8EBFR+jH4yHKSJGkn5F48O/YOE7OrT44eLhbLz686ETedfRwmmvozSgudOH1qBWRZxpnTKuF1O+IGHgBQ6LSjA7GHeV08uwYvb24a0qm1REQ0OkmyLKc223uEdXV1obS0FJ2dnSgpSe/R4LmqbyCEI519WZvGX/YPZW5JTYkHa75nHCHv6w/gjW3NOOf4KsM4cyIiGl1SuX8z85EDClz2rA08AOC7F87AlMoinH18dHbD63HispPGZWBVRESUrRh80LAVuOz4/KkTM70MIiLKEZyARERERJZi8EFERESWYvBBRERElmLwQURERJZi8EFERESWYvBBRERElmLwQURERJZi8EFERESWYvBBRERElmLwQURERJZi8EFERESWYvBBRERElmLwQURERJbKulNtZVkGAHR1dWV4JURERJQscd8W9/FEsi748Pl8AICGhoYMr4SIiIhS5fP5UFpamvAxkpxMiGKhcDiMw4cPw+v1QpKktD53V1cXGhoacODAAZSUlKT1uSmC19kavM7W4HW2Dq+1NUbqOsuyDJ/Ph7q6Othsibs6si7zYbPZUF9fP6Jfo6SkhD/YFuB1tgavszV4na3Da22NkbjOg2U8BDacEhERkaUYfBAREZGl8ir4cLvduOuuu+B2uzO9lFGN19kavM7W4HW2Dq+1NbLhOmddwykRERGNbnmV+SAiIqLMY/BBRERElmLwQURERJZi8EFERESWGlXBx29+8xtMmjQJHo8H8+fPx6pVqxI+fsWKFZg/fz48Hg8mT56M3/72txatNPelcq2ff/55nHfeeaisrERJSQlOPfVUvPrqqxauNnel+jMtvPvuu3A4HJg7d+7ILnCUSPU6+/1+3HnnnZgwYQLcbjemTJmCP/7xjxatNnelep2ffPJJzJkzB4WFhaitrcUXv/hFtLW1WbTa3LRy5UpccsklqKurgyRJeOGFFwb9nIzcC+VR4umnn5adTqf8yCOPyFu3bpVvvvlmuaioSN63b1/Mx+/Zs0cuLCyUb775Znnr1q3yI488IjudTvnZZ5+1eOW5J9VrffPNN8s/+9nP5A8++EDesWOHfMcdd8hOp1P+8MMPLV55bkn1OgsdHR3y5MmT5fPPP1+eM2eONYvNYUO5zpdeeqm8aNEiefny5XJjY6P8/vvvy++++66Fq849qV7nVatWyTabTf7lL38p79mzR161apU8a9Ys+bLLLrN45bnl5Zdflu+88075ueeekwHIf/vb3xI+PlP3wlETfCxcuFC+/vrrDe+bMWOGfPvtt8d8/G233SbPmDHD8L7rrrtOPuWUU0ZsjaNFqtc6lpkzZ8o//OEP0720UWWo1/nqq6+W//u//1u+6667GHwkIdXr/K9//UsuLS2V29rarFjeqJHqdb733nvlyZMnG973q1/9Sq6vrx+xNY42yQQfmboXjoqyy8DAANavX4/zzz/f8P7zzz8fq1evjvk57733XtTjL7jgAqxbtw6BQGDE1prrhnKtzcLhMHw+H8rLy0diiaPCUK/zo48+it27d+Ouu+4a6SWOCkO5zv/4xz+wYMEC/PznP8e4ceMwbdo0fPvb30ZfX58VS85JQ7nOixcvxsGDB/Hyyy9DlmUcPXoUzz77LD71qU9ZseS8kal7YdYdLDcUra2tCIVCqK6uNry/uroaTU1NMT+nqakp5uODwSBaW1tRW1s7YuvNZUO51mb33Xcfenp68JnPfGYkljgqDOU679y5E7fffjtWrVoFh2NU/NMecUO5znv27ME777wDj8eDv/3tb2htbcUNN9yAY8eOse8jjqFc58WLF+PJJ5/E1Vdfjf7+fgSDQVx66aX49a9/bcWS80am7oWjIvMhSJJk+H9ZlqPeN9jjY72foqV6rYU///nPWLZsGZ555hlUVVWN1PJGjWSvcygUwjXXXIMf/vCHmDZtmlXLGzVS+XkOh8OQJAlPPvkkFi5ciIsvvhj3338/HnvsMWY/BpHKdd66dSu+8Y1v4Ac/+AHWr1+PV155BY2Njbj++uutWGpeycS9cFS8PKqoqIDdbo+KoJubm6MiOqGmpibm4x0OB8aOHTtia811Q7nWwjPPPIMvf/nL+Otf/4pzzz13JJeZ81K9zj6fD+vWrcOGDRtw0003AVBukrIsw+Fw4LXXXsPZZ59tydpzyVB+nmtrazFu3DjD0eHHH388ZFnGwYMHMXXq1BFdcy4aynW+5557cNppp+E73/kOAODEE09EUVERzjjjDPzkJz9hdjpNMnUvHBWZD5fLhfnz52P58uWG9y9fvhyLFy+O+Tmnnnpq1ONfe+01LFiwAE6nc8TWmuuGcq0BJePxhS98AU899RRrtklI9TqXlJRg8+bN2Lhxo/Z2/fXXY/r06di4cSMWLVpk1dJzylB+nk877TQcPnwY3d3d2vt27NgBm82G+vr6EV1vrhrKde7t7YXNZrxF2e12AJFX5jR8GbsXjmg7q4XENq4//OEP8tatW+VbbrlFLioqkvfu3SvLsizffvvt8uc//3nt8WJ70Te/+U1569at8h/+8AdutU1Sqtf6qaeekh0Oh/zQQw/JR44c0d46Ojoy9S3khFSvsxl3uyQn1evs8/nk+vp6+aqrrpK3bNkir1ixQp46dar8la98JVPfQk5I9To/+uijssPhkH/zm9/Iu3fvlt955x15wYIF8sKFCzP1LeQEn88nb9iwQd6wYYMMQL7//vvlDRs2aFuas+VeOGqCD1mW5YceekieMGGC7HK55Hnz5skrVqzQPnbttdfKZ511luHxb7/9tnzSSSfJLpdLnjhxovzwww9bvOLclcq1Puuss2QAUW/XXnut9QvPMan+TOsx+Eheqtd527Zt8rnnnisXFBTI9fX18q233ir39vZavOrck+p1/tWvfiXPnDlTLigokGtra+X/+I//kA8ePGjxqnPLW2+9lfD3bbbcCyVZZv6KiIiIrDMqej6IiIgodzD4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJLMfggIiIiSzH4ICIiIksx+CAiIiJL/f+haKJQeCE1/AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(lri, lossi);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "26c6bece",
   "metadata": {},
   "source": [
    "Let's go further and actually determine the $lri$ associated with the lowest loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "25bea6a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.1929)\n"
     ]
    }
   ],
   "source": [
    "# Get index of smallest loss in lossi\n",
    "print(lri[lossi.index(min(lossi))])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e43152e",
   "metadata": {},
   "source": [
    "With this, we have a well-founded confirmation that $0.1$ is within a good region of values to choose for the learning rate.<br>\n",
    "Smaller values have the descent bottom out at a too shallow level, while larger values make the descent too bumpy.<br> \n",
    "Note that now, also, towards the end of the training, learning rate decay is applied.\n",
    "\n",
    "Ok, then let's train properly now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "22b91f66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the parameters once again\n",
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)          # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)        # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)          # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)       # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)           # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "5722b7e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(70000):\n",
    "    \n",
    "    # mini-batch construct\n",
    "    ix = torch.randint(0, X.shape[0], (32,))\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[X[ix]] # (32, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2\n",
    "    loss = F.cross_entropy(logits, Y[ix])\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Learning rate decay to more closely approach the minimum\n",
    "    lr = 0.1 if i < 50000 else 0.01\n",
    "\n",
    "    for p in parameters:\n",
    "        p.data += -lr * p.grad\n",
    "    \n",
    "# print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2674fd85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.305272340774536\n"
     ]
    }
   ],
   "source": [
    "emb = C[X] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 100)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Y)\n",
    "print(loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "47f164aa",
   "metadata": {},
   "source": [
    "We achieved a lower loss than before. But is that good? How do we know?<br>\n",
    "The model itself is small, only $3481$ parameters. Larger models could bring better performance, or bad cases of overfitting.<br><br>How could we detect that?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9bb46047",
   "metadata": {},
   "source": [
    "### Split the dataset\n",
    "\n",
    "The larger the model, the bigger the risk of it memorizing the dataset verbatim. This is called, and we've seen this before, **overfitting**.<br>\n",
    "So, to avoid our performance measurements to be distorted by how well the model may have just memorized instead of generalized,<br>we need to split the dataset into portions for training, validation and testing.<br>The latter ones the model has never trained on, so we can use them to measure the generalization performance of the model.\n",
    "\n",
    "There is `training split`, `dev/validation split`, `test split`.<br> The distribution of the data among these subsets is roughly chosen domain-independently like this: $80\\%$, $10\\%$, $10\\%$\n",
    "\n",
    "Splitting the datset looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c56440ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set:\n",
      "X: torch.Size([182625, 3]) \tY: torch.Size([182625])\n",
      "Validation Set:\n",
      "X: torch.Size([22655, 3]) \tY: torch.Size([22655])\n",
      "Test Set:\n",
      "X: torch.Size([22866, 3]) \tY: torch.Size([22866])\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def build_dataset(words):\n",
    "    block_size = 3\n",
    "    X, Y = [], []\n",
    "    \n",
    "    for w in words:\n",
    "        context = [0] * block_size\n",
    "        for ch in w + '.':\n",
    "            ix = stoi[ch]\n",
    "            X.append(context)\n",
    "            Y.append(ix)\n",
    "            context = context[1:] + [ix] # crop and append\n",
    "    \n",
    "    X = torch.tensor(X)\n",
    "    Y = torch.tensor(Y)\n",
    "    print('X:', X.shape, '\\tY:', Y.shape)\n",
    "    return X,Y\n",
    "\n",
    "random.seed(42)          # for reproducibility\n",
    "random.shuffle(words)    # words is just the bare list of all names, from wayyy above\n",
    "n1 = int(0.8*len(words)) # index at 80% of all words (rounded for integer indexing)\n",
    "n2 = int(0.9*len(words)) # index at 90% of all words (rounded for integer indexing)\n",
    "\n",
    "print('Training Set:')\n",
    "Xtr, Ytr = build_dataset(words[:n1])     # The first 80% of all words\n",
    "print('Validation Set:')\n",
    "Xdev, Ydev = build_dataset(words[n1:n2]) # The 10% from 80% to 90% of all words\n",
    "print('Test Set:')\n",
    "Xte, Yte = build_dataset(words[n2:])     # The 10% from 90% to 100% of all words"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9f7ad22d",
   "metadata": {},
   "source": [
    "Before we play around with the model architecture, let's first see how well the current model performs on the validation set and the test set when trained on the training set only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "42dc9158",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3481 parameters\n"
     ]
    }
   ],
   "source": [
    "# Reset the parameters once again\n",
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)          # 27 characters, 2 dimensions each\n",
    "W1 = torch.randn((6,100), generator=g)        # 3 characters times 2 embedding values as inputs to 100 neurons\n",
    "b1 = torch.randn((100), generator=g)          # 100 biases added to the 100 neuron outputs\n",
    "W2 = torch.randn((100,27), generator=g)       # 100 neuron outputs as inputs to 27 output neurons\n",
    "b2 = torch.randn((27), generator=g)           # 27 biases added to the 27 output neurons\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "print(sum(p.nelement() for p in parameters), 'parameters')\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "d3365563",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(40000):\n",
    "    \n",
    "    # mini-batch construct\n",
    "    ix = torch.randint(0, Xtr.shape[0], (32,))\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[Xtr[ix]] # (32, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2\n",
    "    loss = F.cross_entropy(logits, Ytr[ix])\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Learning rate decay to more closely approach the minimum\n",
    "    lr = 0.1 if i < 30000 else 0.01\n",
    "\n",
    "    for p in parameters:\n",
    "        p.data += -lr * p.grad\n",
    "    \n",
    "# print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "1d4493d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.34187912940979\n"
     ]
    }
   ],
   "source": [
    "# Validation loss\n",
    "emb = C[Xdev] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 100)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Ydev)\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6658b0ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.338966131210327\n"
     ]
    }
   ],
   "source": [
    "# Test loss\n",
    "emb = C[Xte] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 100)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Yte)\n",
    "print(loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ddffb8dd",
   "metadata": {},
   "source": [
    "We can see that validation and test loss are about equal. This is good. It means that the model is not overfitting.<br>\n",
    "The model is not powerful enough to memorize the training set, as it's too small.<br><br>**We underfit.**<br><br>\n",
    "This is a sign for us that we are in fact able to increase the model size without risking overfitting too quickly.<br>\n",
    "\n",
    "If we want to really see the impact of splitting the dataset, we need to start playing around with the total parameter count of the model.<br>\n",
    "We can do this by changing the width of the hidden layer. \n",
    "\n",
    "Let's try $300$ instead of $100$ neurons:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "9f749e77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10281 parameters\n"
     ]
    }
   ],
   "source": [
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27,2), generator=g)\n",
    "W1 = torch.randn((6,300), generator=g)\n",
    "b1 = torch.randn((300), generator=g)\n",
    "W2 = torch.randn((300,27), generator=g)\n",
    "b2 = torch.randn((27), generator=g)\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "print(sum(p.nelement() for p in parameters), 'parameters')\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "54727424",
   "metadata": {},
   "outputs": [],
   "source": [
    "lossi = [] # list of losses per mini-batch\n",
    "stepi = [] # list of steps (mini-batches)\n",
    "\n",
    "for i in range(180000):\n",
    "    \n",
    "    # mini-batch construct\n",
    "    ix = torch.randint(0, Xtr.shape[0], (32,))\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[Xtr[ix]] # (32, 3, 2)\n",
    "    h = torch.tanh(emb.view(-1, 6) @ W1 + b1) # (32, 100)\n",
    "    logits = h @ W2 + b2 # (32, 27)\n",
    "    loss = F.cross_entropy(logits, Ytr[ix]) \n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    lr = 0.1 if i < 60000 else 0.05 if i < 120000 else 0.01\n",
    "    \n",
    "    for p in parameters:\n",
    "        p.data += -lr * p.grad\n",
    "    \n",
    "    # Loss per mini-batch tracking\n",
    "    stepi.append(i)\n",
    "    lossi.append(loss.item())\n",
    "    \n",
    "#print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "104359f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6QElEQVR4nO3deXQUVcL+8aeB0ARIAgGysYSI4EIQBZRFZBOBiCuOI+IovDq8OoqjP3QQXEZcRhgdHWde9w1x3EcRURAMssq+E8IWJEAgCYGQPWS/vz9CmjTZIalOqO/nnD4nXXW76t6uhn761r1VDmOMEQAAgEUaeboCAADAXggfAADAUoQPAABgKcIHAACwFOEDAABYivABAAAsRfgAAACWInwAAABLNfF0Bc5UVFSk+Ph4+fj4yOFweLo6AACgGowxysjIUEhIiBo1qrxvo96Fj/j4eHXs2NHT1QAAAGchLi5OHTp0qLRMvQsfPj4+koor7+vr6+HaAACA6khPT1fHjh1d3+OVqXfho+RUi6+vL+EDAIAGpjpDJhhwCgAALEX4AAAAliJ8AAAASxE+AACApQgfAADAUoQPAABgKcIHAACwFOEDAABYivABAAAsRfgAAACWInwAAABLET4AAICl6t2N5epKQWGRXpy/S5I0NeJiNfNq7OEaAQBgT7bp+Sgy0serD+jj1QeUV1jk6eoAAGBbtgkfAACgfiB8AAAASxE+AACApQgfAADAUoQPAABgqRqFjxkzZujKK6+Uj4+PAgICdMstt2jPnj1uZSZMmCCHw+H26NevX61W+lwZ4+kaAABgXzUKH8uXL9dDDz2ktWvXKjIyUgUFBRoxYoSysrLcyo0aNUoJCQmux4IFC2q10mfD4fB0DQAAgFTDi4wtXLjQ7fmsWbMUEBCgTZs2adCgQa7lTqdTQUFBtVNDAABwXjmnMR9paWmSJH9/f7fly5YtU0BAgLp166aJEycqKSnpXHYDAADOI2d9eXVjjCZPnqyBAwcqPDzctTwiIkK33367QkNDFRsbq2eeeUbDhg3Tpk2b5HQ6y2wnNzdXubm5rufp6elnWyUAANAAnHX4mDRpkrZv365ff/3Vbfkdd9zh+js8PFx9+vRRaGio5s+frzFjxpTZzowZM/Tcc8+dbTUAAEADc1anXR5++GHNmzdPS5cuVYcOHSotGxwcrNDQUMXExJS7ftq0aUpLS3M94uLizqZKAACggahRz4cxRg8//LC+++47LVu2TGFhYVW+Jjk5WXFxcQoODi53vdPpLPd0TJ1iqi0AAB5To56Phx56SJ9++qk+//xz+fj4KDExUYmJiTp58qQkKTMzU48//rjWrFmjAwcOaNmyZbrxxhvVtm1b3XrrrXXSgOpipi0AAPVDjXo+3n77bUnSkCFD3JbPmjVLEyZMUOPGjRUVFaVPPvlEqampCg4O1tChQ/XVV1/Jx8en1ioNAAAarhqfdqmMt7e3Fi1adE4VAgAA5zfu7QIAACxF+AAAAJYifAAAAEvZMnwY5toCAOAxtgwfAADAc2wTPhwOrvQBAEB9YJvwAQAA6gfCBwAAsBThAwAAWIrwAQAALGXL8FHFVeIBAEAdsmX4AAAAnmOb8MFEWwAA6gfbhA8AAFA/ED4AAIClCB8AAMBShA8AAGApW4YPZtoCAOA5tgwfAADAc2wTPripLQAA9YNtwgcAAKgfCB8AAMBShA8AAGApwgcAALCULcOH4ba2AAB4jC3DBwAA8BzbhA8Hc20BAKgXbBM+AABA/UD4AAAAliJ8AAAASxE+AACApWwZPphoCwCA59gyfAAAAM8hfAAAAEsRPgAAgKUIHwAAwFKEDwAAYCnCBwAAsJQtwwc3tQUAwHNsGT4AAIDnED4AAIClbBU+HA5P1wAAANgqfAAAAM8jfAAAAEsRPgAAgKUIHwAAwFK2DB9GXOgDAABPsWX4AAAAnmOr8MFMWwAAPM9W4QMAAHge4QMAAFiK8AEAACxF+AAAAJayZ/hgpi0AAB5jz/ABAAA8xlbhw8FtbQEA8DhbhQ8AAOB5hA8AAGCpGoWPGTNm6Morr5SPj48CAgJ0yy23aM+ePW5ljDGaPn26QkJC5O3trSFDhig6OrpWKw0AABquGoWP5cuX66GHHtLatWsVGRmpgoICjRgxQllZWa4yL7/8sl577TW98cYb2rBhg4KCgnTdddcpIyOj1isPAAAaniY1Kbxw4UK357NmzVJAQIA2bdqkQYMGyRij119/XU899ZTGjBkjSZo9e7YCAwP1+eef6/7776+9mp8DZtoCAOA55zTmIy0tTZLk7+8vSYqNjVViYqJGjBjhKuN0OjV48GCtXr263G3k5uYqPT3d7QEAAM5fZx0+jDGaPHmyBg4cqPDwcElSYmKiJCkwMNCtbGBgoGvdmWbMmCE/Pz/Xo2PHjmdbpSox0RYAAM876/AxadIkbd++XV988UWZdWdeT8MYU+E1NqZNm6a0tDTXIy4u7myrBAAAGoAajfko8fDDD2vevHlasWKFOnTo4FoeFBQkqbgHJDg42LU8KSmpTG9ICafTKafTeTbVAAAADVCNej6MMZo0aZLmzJmjJUuWKCwszG19WFiYgoKCFBkZ6VqWl5en5cuXa8CAAbVTYwAA0KDVqOfjoYce0ueff67vv/9ePj4+rnEcfn5+8vb2lsPh0KOPPqqXXnpJXbt2VdeuXfXSSy+pefPmGjduXJ00AAAANCw1Ch9vv/22JGnIkCFuy2fNmqUJEyZIkqZMmaKTJ0/qwQcfVEpKivr27auff/5ZPj4+tVLh2mCYawsAgMfUKHyYanxrOxwOTZ8+XdOnTz/bOgEAgPMY93YBAACWslX4qGC2LwAAsJCtwgcAAPA8wgcAALAU4QMAAFjKluHDcF9bAAA8xpbhAwAAeA7hAwAAWMpW4cMh5toCAOBptgofAADA8wgfAADAUoQPAABgKVuGD+5qCwCA59gyfAAAAM8hfAAAAEvZK3ww0xYAAI+zV/gAAAAeR/gAAACWInwAAABL2TJ8MNMWAADPsWX4AAAAnkP4AAAAlrJV+GCmLQAAnmer8AEAADyP8AEAACxF+AAAAJayZfgw3NYWAACPsWX4AAAAnkP4AAAAliJ8AAAAS9kqfDi40AcAAB5nq/ABAAA8j/ABAAAsZcvwwUxbAAA8x5bhAwAAeA7hAwAAWIrwAQAALGWr8OEQc20BAPA0W4UPAADgeYQPAABgKcIHAACwFOEDAABYivABAAAsRfgAAACWslX44K62AAB4nq3CBwAA8DzCBwAAsJQtwwd3tQUAwHNsGT4AAIDnED4AAIClCB8AAMBStgofzLQFAMDzbBU+AACA5xE+AACApWwZPoyYawsAgKfYMnwAAADPIXwAAABLET4AAIClahw+VqxYoRtvvFEhISFyOByaO3eu2/oJEybI4XC4Pfr161db9QUAAA1cjcNHVlaWevbsqTfeeKPCMqNGjVJCQoLrsWDBgnOqZG1xOLjSBwAAntakpi+IiIhQREREpWWcTqeCgoLOulIAAOD8VSdjPpYtW6aAgAB169ZNEydOVFJSUoVlc3NzlZ6e7vaoa9zVFgAAz6n18BEREaHPPvtMS5Ys0auvvqoNGzZo2LBhys3NLbf8jBkz5Ofn53p07NixtqsEAADqkRqfdqnKHXfc4fo7PDxcffr0UWhoqObPn68xY8aUKT9t2jRNnjzZ9Tw9PZ0AAgDAeazWw8eZgoODFRoaqpiYmHLXO51OOZ3Ouq4GAACoJ+r8Oh/JycmKi4tTcHBwXe8KAAA0ADXu+cjMzNS+fftcz2NjY7V161b5+/vL399f06dP12233abg4GAdOHBATz75pNq2batbb721Vit+NphoCwCA59U4fGzcuFFDhw51PS8ZrzF+/Hi9/fbbioqK0ieffKLU1FQFBwdr6NCh+uqrr+Tj41N7tQYAAA1WjcPHkCFDZCqZq7po0aJzqpAVmGkLAIDncG8XAABgKcIHAACwFOEDAABYivABAAAsZa/wwVxbAAA8zl7hAwAAeJwtw0dlU4UBAEDdsmX4AAAAnkP4AAAAliJ8AAAASxE+AACApWwVPphpCwCA59kqfAAAAM+zZfhgoi0AAJ5jy/ABAAA8h/ABAAAsRfgAAACWInwAAABL2Sp8OBxMtgUAwNNsFT4AAIDnET4AAIClbBk+DBf6AADAY2wZPgAAgOcQPgAAgKUIHwAAwFKEDwAAYClbhQ8u8wEAgOfZKnwAAADPs2n4YK4tAACeYtPwAQAAPIXwAQAALEX4AAAAliJ8AAAAS9kqfDDTFgAAz7NV+AAAAJ5ny/DBXW0BAPAcW4YPAADgOYQPAABgKcIHAACwFOEDAABYylbhw8FtbQEA8DhbhQ8AAOB5tgwfzLQFAMBzbBk+AACA5xA+AACApQgfAADAUoQPAABgKVuFDybaAgDgebYKHwAAwPNsGT64qy0AAJ5jy/ABAAA8h/ABAAAsRfgAAACWInwAAABLET4AAIClbBU+HFzoAwAAj7NV+ChhuK8tAAAeY8vwAQAAPKfG4WPFihW68cYbFRISIofDoblz57qtN8Zo+vTpCgkJkbe3t4YMGaLo6Ojaqi8AAGjgahw+srKy1LNnT73xxhvlrn/55Zf12muv6Y033tCGDRsUFBSk6667ThkZGedcWQAA0PA1qekLIiIiFBERUe46Y4xef/11PfXUUxozZowkafbs2QoMDNTnn3+u+++//9xqCwAAGrxaHfMRGxurxMREjRgxwrXM6XRq8ODBWr16dbmvyc3NVXp6utsDAACcv2o1fCQmJkqSAgMD3ZYHBga61p1pxowZ8vPzcz06duxYm1U6A3NtAQDwtDqZ7eI444Iaxpgyy0pMmzZNaWlprkdcXFxdVOmM+tT5LgAAQAVqPOajMkFBQZKKe0CCg4Ndy5OSksr0hpRwOp1yOp21WQ0AAFCP1WrPR1hYmIKCghQZGelalpeXp+XLl2vAgAG1uSsAANBA1bjnIzMzU/v27XM9j42N1datW+Xv769OnTrp0Ucf1UsvvaSuXbuqa9eueumll9S8eXONGzeuVisOAAAaphqHj40bN2ro0KGu55MnT5YkjR8/Xh9//LGmTJmikydP6sEHH1RKSor69u2rn3/+WT4+PrVXawAA0GA5jKlfwy/T09Pl5+entLQ0+fr61uq2+7y4WMczc/XTI9fokuDa3TYAAHZWk+9vW93bhbvaAgDgebYKHyXqV18PAAD2YsvwAQAAPIfwAQAALEX4AAAAliJ8AAAASxE+AACApWwVPphpCwCA59kqfJQwYq4tAACeYsvwAQAAPIfwAQAALEX4AAAAliJ8AAAASxE+AACApQgfAADAUrYKH45TF/rgrrYAAHiOrcIHAADwPMIHAACwFOEDAABYivABAAAsRfgAAACWInwAAABL2Sp8OOTwdBUAALA9W4UPAADgeYQPAABgKcIHAACwFOEDAABYivABAAAsRfgAAACWslX44K62AAB4nq3CBwAA8DzCBwAAsBThAwAAWIrwAQAALGWr8JGcmSdJyiss8nBNAACwL1uFj5LQ8ebSfR6uCQAA9mWr8FFiye4kT1cBAADbsmX4AAAAnkP4AAAAliJ8AAAASxE+AACApQgfAADAUoQPAABgKcIHAACwFOEDAABYivABAAAsRfgAAACWInwAAABLET4AAIClCB8AAMBStg0fX6w/pMIi4+lqAABgO7YNH9PmROmrDXGergYAALZj2/AhSVvjUjxdBQAAbMfW4QMAAFiP8AEAACxF+AAAAJYifAAAAEvZOnz8sC2B6bYAAFis1sPH9OnT5XA43B5BQUG1vZtacTK/UF9uOOTpagAAYCt10vPRvXt3JSQkuB5RUVF1sZtasea3ZE9XAQAAW2lSJxtt0qTe9nacyeFweLoKAADYSp30fMTExCgkJERhYWEaO3as9u/fX2HZ3Nxcpaenuz2sRPQAAMBatR4++vbtq08++USLFi3S+++/r8TERA0YMEDJyeWf3pgxY4b8/Pxcj44dO9Z2lSq1IuaYihh0CgCAZRzGmDr95s3KylKXLl00ZcoUTZ48ucz63Nxc5ebmup6np6erY8eOSktLk6+vb63WpfPU+eUuv/Oqjpox5rJa3RcAAHaSnp4uPz+/an1/1/lU2xYtWqhHjx6KiYkpd73T6ZSvr6/bw2pfrOcGcwAAWKXOw0dubq527dql4ODgut7VOdmXlOnpKgAAYAu1Hj4ef/xxLV++XLGxsVq3bp1+97vfKT09XePHj6/tXdWq4a8t93QVAACwhVqfanv48GHdeeedOn78uNq1a6d+/fpp7dq1Cg0Nre1dAQCABqjWw8eXX35Z25u0zKp9x9WxdXOti01WI4dDt/Xu4OkqAQBw3qmTi4w1VHd9sM7teUSPIDVvylsEAEBtsvWN5aqSX8D1PwAAqG2Ej0p8vPqA/v3L6SnCadn53AUXAIBzxDmFSvxz8V5J0vU9gtWkkUND/rFM4e191bNDK117SYCGXRzo4RoCANDw0PNRDZm5Bfp6Y/GFyHYcSddn6w7p3o83SpJSs/PcekMycwt094fr9OX6Qx6pKwAA9R3hoxpe/XmP3lr2W5nle49m6PLnIzXu/bWuZe+t2K+VMcc1dU6UlVUEAKDBIHxUw8qY4+Uu/3pDcW/IutgTrmXpJ/MtqRMAAA0V4aOWORyerkH9tvq343ros81KysjxdFUAAB7CgNNz8MGvsWWWOXQ6fcxYsEuXdWil0ZfV7/vaWGnc+8XXUiksMnrn7t4erg0AwBPo+aglscezJElGpwefvrtivx76fLMkKSe/UEVnTNNdtidJv+w6WuE2cwsKXX8XFhnlFxa5rY87kV1mmw1FfNpJT1cBAOAhhI9aMvQfyzTlm22atepAmXVp2fm69K8LdcGTCzT0H8uUnJmrnPxCTZi1QffN3qi0csaJrIw5poueXqg3l+6TJI18fYX6z/jFFUD+s/agrnl5qZ78joGt1bX/WKbW7k9WRk6+fvf2an1UTs8VAKDuET5q0dcbD5e7fOmeJJV0UMQez9Lkr7epoFSPRUZO2fAx7dRsmVcW7ZExRvuSMnU8M08HTvWwvPbzHknSlxviZMy5937k5Bfq+61HlJyZW2GZ345llqnrtrhUjfjnci3fe6xG+zuYnK3svIKzqmt1nXlBuGGvLtfY99bqye92aOPBFD3/48463T8AoHyEDwukZOe5PV++95h+LTWDxpjiUywr9h7TVxvKXh/k+63xrr8djuKgkJJ9OgSETVugm974Vd9sOqzZqw8obNp8JaXn6Om5Ueo8db6W7C57ascYo+OlgsbMn3brkS+36o73iqcNbz6Uoi2HUlzro+PTdO2ryzVg5hJJ0nM/ROuhzzfr9nfXaO/RTI3/aL1+jTmuUa+v0La41Crfk7ST+Rr2j+Vl6lRwxqmlM1U3sJzIylPvFyP1xDfby6zbfDClnFecn3LyC7V0T5Jy8gurLgwAFiF8WOC5H8r+wn7g002uv//1S4wuenqh7vlovZ74tjgwJGWcDgaPfrXV9bfD4dC9H28os73th9P0+H+36dl50TJGuuqlX/Tp2uIgc+/HG5WSdToA5RUU6Q8frlOfFxfrjSUxrl4PSdqXlKmTeYUa89Zq3frWav24PV73frxB/z3Vq5ORU/zlP2vVAc3fnqC8gtNh4Q8frtPuxAzd89H6ar0vienuM14e+HST+vxtsY6kntSJrLwy5T9Zc0CX/nWR/nvqgm+lbY1L1dFS25u1Klap2fn6amP1eoZW7zuuV3/eU2742XEkTVfPXOJ6j6TiXqA/zt6o7YdTq9y2J/3lm+36n1kbXD1p55Po+DSlZTO1HWiIHKY2+uxrUXp6uvz8/JSWliZfX99a3XbnqfNrdXsNiZ+3l+4bGKbXIvdWWfbKzq214UDFvQOv/b6nJn+9rdJtHJg5utzlZx6D2BnXy+FwKK+gSN2e/slt3a7nR8m7aWNJ0r6kDA1/bYVr3YAubdQt0EfTb+qu6Pg0jf73r679zvhpl95dvt9Vtp2PUx/c00c3v7lKkhTk28wVfErqWVKvv9/WQ3dc2cmtHoNfWaqDydlu5a95eYniTpystK31Qen3uz7Xs6Y2HDih299ZI59mTRQ1feQ5by8pPUePf7Nd3UN8dUHbFup3QRt1aO0th4Vz5/clZSq0TXN5NeY3IRqmmnx/8ym3ibST+dUKHpIqDR6SqgweUvGXXsmX/bxt8Yr410rXjKDSZq06oPdX7C8TPCTpSOpJV69F6eAhSat/S9bHqw/ogf9s0oZSF3nrPHW+W/CQpGMZuRr73umr0JbucZm75YhO5p0+JfH5OvfTXoVFRqmlfl2/+vMepWbnuYKHJD329Tb9cfbGWhl7c6aiIqPo+LQqb2iYlJ6jn6IS6tWND0/mFZZ5T7JyCzRvW7wyc8uePkvOzHWNaarK4lOzxEp64s7Vcz/s1Iq9x/T2st/0l2+265qXl+r9lfurfmEtmbvliIa/trzcXk2cm9X7jmv6vGhbnXrMyi3Qj9vL/3dWX3CdD9SZbXGp+vvC3Xr71KXpJ8wqezqmskGfv393TbmnX0pbGJ1YZgpyeU5W8B9P6VNakrTtcJqrt6BbYEvtPZrptv7/luzT/y3Z57bs283Fp6R+O5alCwNalrufvIIizVoVq0Hd2qmg0KiTf3P5Nfeqst5X/32JEtJyNL5/qO4dGCavxo0U0sq7TLlrX12ujNwCTb/xUo0f0FkHkrPVuU3zSrf9U1SC4tNydN/AsCrrUZWj6Tl66rso3dO/swZ1a6cjqSd19cwlGnJRO338P1e5yk3+eqsWRR/V8EsC9MH4K9220fvFxZKkdU9eq0DfZpXv8Cwy1qdrD+q1yL1aOWWoWjhP/9cXHZ+m+VEJZcq/tGC3/ndQl5rvqArGmDI9KrNWFc+8quhqyvXF+tgTWrH3mP58bVc1bdIwfruO+6D42kKtmnvp0eHd6nx/xzNz9fwPO3XnVZ3Uv0ubOt9feaZ8u13ztxd/pu8fdIGmXX+JR+pRmYbx6UGD9Xape+KUnLqorqqCR4lfdifVaLvVdWbwqErpXoeCwiJN/nqrfo5OlCR98Ot+zfhptyL+tVI3vvGrej7/s+s1d3+4Tje/uUppJ/P1x9kbtDLm9MyhhLTiXprZaw5q8CvLNGDmEhljlHYyX0dST/e+ZJz6hTP9h52656P1GvqPZfrXLzFu9Zv89VZllfol9KfPNuuFH3cqOj5NY95apdvfWV1p+3LyC7VwR2K5v6ae+m6HFu9Kco33+XzdQUnSsj3us6AWRRf3WCzeVfEx25mQrp3x6brno/WKOpxWbpl3V9SsV6KwyOjpuTt0IitPN/zfr27rSk7Zna24E9nacSRNS3cnac1vyRVee2fetnh1njpfYdMWaN62eM1YsEsP/GeTioqMW5bak5ih8R+t157EDNeyzNwCzdl8uMyAa2OM4k5k6+WFu2t01eCXFuzSq6dmzJVW1YDu37+7Rm8s3af/rD3oWrYy5phueXOV9iRmKL+wSE98s11DXlnqVv9zVVBYpC/WH9Jtb68+66sjHzpRs/9/ztaz30dr3rZ43Xnqnl+p2Xn6Z+ReHUyuXq9ebSgJHlLN/61YhZ4PoJYUGaMVe4+5Dbids/mIDswcXe6X6IKoBB1OyXb92u35XHEgWbwrSb8+MVTr9p8o8xqpeHZTifVPXquAM3oJSrb3+mL38DFn8xHFHs/Sdw9e7fYFuXBHojYfSpUkrdufrL4XuP9ayy8s0p7EDD3/w06tP3BC4e191aN9K93Vt5PC2/tJkhJKXTTug5X79ebS06Fz08ET6h3qr3nb4t22m1dQpH/8vEfX9wjW5R1buZb/z6zTpx7W/pasvX+LkCR9vCpW08sZvN156nzdekV7LYhKUPtW3vr2TwO0MyFdCWk5uq1Xex1OOek6TSOdviDgpoMnFHu88i+kj1fFau7WeI3p1V4frIzVv++8wq2uknTNy0vLfe2BmaNljNFXG+LUqrmX/vzFFte60n9viUtR6bNTI18vPsW4fO8x1zidiH+tUNyJk5r89TbXsj2JGbrrg3WuWWsbD6bo6/v7u9XhZF6hmnk1cutpOZqeo/dOfSE9NPRCNfMqHlf1U1SC/vTZZk0ZdZH+NLiL6zUZOflyNmns1tPxwo87dVGgjwZ2bau7Pyz+vE/8ZKPu7heqr04NCB/73hpt+euIMu9LSc/PgeNZcjik0DYtJEmLohO1JzFDDw+7sEzP0JB/LNPhlOLP2MyfduuZ0ZcqK69AHVpX3rt3Nv4ZuVdHUk/qld9d5laPhLSTWvNbsm64LMT1XhhjtOlgioL8mrnqEpfi/pma+m2UFkYnataqWG2fPlJpJ/Pl512217OoyKhRI0eFzytzND1HAT5OS8conSvCB1BL7vt4g+LTyv4qq2ig84Ofba5wWwP/Xv4X2pkmfb5FDw6t/qmBLYdS9fWGOE359vQU5NKnkd5fGaux769ViJ+3PppwpS4K8tG9H29wOx2w40i6dhxJ1xfrD2nllKE6kJyl6Ph01/oX5+9y2+dtb6/RFZ1aacupgFOiZJzPeyv269cnhpZb37xTp9S+WH+o3OBR4rstxTOR9h/P0hUvRLqWP/7f8scnncjK021vr6lweyVK9rn11PTxW06NYyrx1xsurfC19328QS2cTcqErjO9vjhGUUfK7+F5acEuOSS3MUZS8RiuJ77d7jZdfsMB97B6KDlbg15ZqpHdA/Xu3X1cy0vPUHvos826NMRX3QJ99PCpQPTywj2atzVeF7RroRdv6aFep97P2BnXu23/Dx+ucxvEfCIrz/U+SVJKdr42HUxRj/Z+atqkkTYdPKHb3l6jls4m+n7S1br21eKp9ntfjFB2XoHu/0/xDMArOrVS/wva6Pkfd2pAl7YaFR7kCh6SlJlT4DrGG58errYtncrJL9S+pEx1D/F1+wIuPZ4rt6BIS3cnqXuIr5xNGsuvuZcycwu0cEeirrskUH7NvVRYZFw9hq2be2nKqIt1MDlLT323w3UD0fjUk5o0rKsk6ZddSfrjJxtd+3jh5u5u79GvMce18FTvZ3pOgSJ3HtXETzbqjwPD9PQNlyolK09ztx5xzYh84ebuurt/Zy2IStDkr7fq1dsvV6/QVjqanlsm9JaYty1ef/5ii4ZdHKCPJlxZ5eUK6gtmuwBAA/L06EvKBLwS7Vt5a2rExbqxZ4jb/3cTBnTWtOsvVm5BkdKy8yvsrTnTlFEX6eWFxadnfJxNXKf3Smx+5jpXOGnpbKI+nVuXOdV28+UhGtytXYUD1X99Yqhb2H7ld5fpi/WHXL1x/32gv25/53RQLD0b741xVygiPFjXvbZc+49n6YWbu2v2moPqf0Eb3XlVJ13/75UVtu2Cdi20/9jpUyE3Xx6ix667SINeqfq9GX1ZsP5v7BUaP2t9mXE6Pdr7VRgmO7dprgOlZs6V9520/C9DNPiVZWWWTxl1kcb376wv1h/S3xfuVn6h0YNDuuitUqe2/3fQBa5erdImDOisx0depNz8Qvm3aFpnPSQ1+f4mfADAeWZMr/aas/lI1QWrcOYXdGVaNG2srDz7zCh5/ubu+uv30Z6uxllb8OdrdGlI7X7HMtUWAGysNoKHpGoHD0m2Ch6SGnTwkFRpr5AVbBU+gqqavgcAAOqcrcIHAADwPFuFjxduCfd0FQAAsD1bhY9Lgn08XQUAAGzPVuGjIV2ABQCA85WtwkeIHwNOAQDwNFuFD3o+AADwPFuFDwAA4Hm2Cx8bnhru6SoAAGBrtgsf7Xycnq4CAAC2ZrvwUZ7hlwTolstD3O7QCAAA6kYTT1fAk1b8Zag6tPZWo0anB6LOvvcqTfpsc5m7N55p1dRhat/KW8YYrd1/QiGtmunTtQf1/srYuq42AAANmi3Dx9LHhygjJ1+d2jQvs25wt3aKem6kjDHafjhNr0buVZCvU19vPOxWzutUYHE4HOrfpY0k6anRl2rb4TStjz3hKhcRHqQHh1yozm2bq8f0n8vsb9Gjg5SanaeU7DxtjUvTO8t/K1NGkm69or2+21I7N4sCAMCTbBk+wtq2qLKMw+FQz46t9Mm9V2n/sUy38HF5x1YKqOAmdf8ae7le+HGnRvcIUXJWrsb06qCWzuK3+ev7++u/G+N03aWBOnQiW7kFRboo6PRVV0eFB5cbPj64p4+GXxp4VuGjd2hr7T2aoYycyntySpv70NW65c1Vruf3DQzTh79Wv0fnlstDNHdrfI3qCQCwD1uGj5q6oF1LfffgALVt6VRH/7K9JaUF+3nrrbt6l7vuqjB/XRXmX+nr3727t+7/zyYNvyRQxzNz9crvLlPXwPIvC//PO3rqnWX71dHfW4t3JbmW3zcwTF6NG+nR4V3VzKuxcvILdfEzC13r10wbpuy8QoX6N1dBkXFbd0HbFurZwc9tP1MjLtYjw7vqsnJ6bkq0b+Wtzyf2VW5BkboGtNTR9Fyti01WkSm//EWBPtpzNEMjuwdqUfTRMuuHXNROb47rpfjUk7runysq3G91tW7upU5tWsgYo71HM9Q9xE+bDqa4lel3gb86+TfX5OsuUr8Zv7iWh7Vtodjj1b+1eHWM7x+q2WsO1uo2AaChIHxU0xWdWluyn5Hdg7TvbxFq0rjsWOAXbu6uVyP36s1xvXR5x1Zq4WyiW6/oIEmavz1Bz87boTfH9VLfC9q4va6ZV2PX37dcHqJgP2/X8yaNpZ3Pj9SxjFyFtPJWI4dDDofD7QvXq3EjeZWqzxWdWqlPaGt1addSY3p10C+7jqrvBW3k36Kpq8znE/tKkr7ZdFh/+Wa7pt94qbyaNNJT3+3QG+Ou0A2XhUiSEtNyXOHjp0euUWGRUdfAlnI2Ka5z10Af/fbS9Vq3P1n+LZsqMvqo7rsmTJc/H6m8giK3dsb8LUJdn/rJbdmeF0e5tlWiqMi4jfOpzN39Ql03JPzPmgPyaeYlZ5NG+tNnm8uU7ejvrbgTJ13Pm3k1Uk5+UZlycx4coJbOJm7h46owf91yeXs9+V2Ua9m2Z0eo53MVB77Snh59iZxejZWTV6i/LdjlWt66uZdSsvOrtY3yPHZdN70aufesXw8A5XEYYyr4beoZ6enp8vPzU1pamnx9fT1dnXrHGFPhlVorWxd1OE3fbIrTo8O7qXWpkFCRuz5Yq1X7kiXJNQtoZcwxvb3sN80Y00Ohbao+dVWe3IJCtzCQW1Coi54u7nmJ+VuEW8ipzKaDKfrr9zv0zA2X6tO1B9W2pVPTb+ouY4yKjNS4kaPS96MqEf9aqV0J6Vo5ZWi5vV1/m79T76+MVftW3jqSelIBPk6tf2q4rp65REdST+rl312moRcF6LN1B3UoOVvjB3TWzW+u0ri+nfTSrT0kSW8u3adXFu2RJC3/yxCFtmmhzlPnu/Zx5uyrZXuS9PeFezSye6DeW7FfXdq11CPXdtXwSwPdyv2wLV7PzovWG+Ou0KXBvrr8+Ui39dddGihnk0Y6npmrj//nKi2KTtQVHVvLz9tLPZ8vDjsv3NxdN1/RXr7NvNzq1P+CNnpgSBfN/Gm3Xr29p1o199KAmUvUN8xfOxPSNap7kC5o11J/X7hbktTJv7lGhQfpvRX71aZFU736+56aMGuDJGnllKG65uWlkqQRlwbqnT/01vM/7tTHqw9UeFzOPCXYuU1zZeYWasqoi9S0cSO9v3K/ouPTJUmf/7GvBlzY1q3+pb1wS7huuixEfs29XMuW7D6qxo0a6fstRzTnjNOcE68JU0grbz33w07X8dl44IQKi4zueG+tJKmRQ9ry1xFatCNRU77d7vZ6n2ZNtP7J4fJu2ljHMnLVzsepoiKjxbuO6n//s0lS8UD2q2cu0cAL22pkeJCembujTL0HXthWv+47XuF7VOKe/qH65IzetVVThynEr5nCpi0oU37Pi6OUW1Dk6uG8qWeI5m07ffr0mRsu1SdrDuhgcrYkydursabfdKme+LY4MG9+5jr1euH0Z+3R4V31+uKYCut3VZi/a3zcL48NVmp2nnLyi3TXB+uqbNvZKnlfvRo5NHVOVNUvqIYbLgvWj9sTqiz3r7GX65Evt7qe+zZrovRSp8MnXhNW7QkL/xp7ufpd0EYxRzN19YVtyhxPP28vpZ0s/0fH+qeu1cHkbF3YrmW1vgtqoibf34QPlCs+9aSembtD9w4M09UXtq3TfR3LyFUjh9SmZf25BktBYZEycwvUqnnF/zjzCorLfLLmgG7r1cEVUgoKi8rtuapI6ZB0Mq9QWw6l6Kow/xpto7Jt7j2aoX1JmQrya6Yv1h3SlFEXV3i9m9LhrURSeo6ueukXjexeHBCqE+iOpufo282HNfbKTmretLHmbYvXoK7ttP94psa9X/zlcmDmaOUVFKlxI4drf8YY3f3herVwNtY7f+it6Ph0bT6Uoj/0DZXDUTwW66eoBP3ps80K8HFq0aOD1Kq5l6tOeQVF2peUqaZNGunCgJaS5BY+/ufqzpq16oB6tPfTDw8PrLQNh5KzNeiVpfJv0VTL/zJEPs28lJNfqLs+WKe+Yf6aMupiV9lpc7bri/Vx+n/Du+mR4V1ljNHq35J1SbCvRv97pRLSctx6/KqrdN0DfZ16/uZwjewepBNZeWp9KjSVFySu7xGkN8f1UnJWnvq8uNi1vCTQ/rg9Xk/OidIb43ppZ0K6enZo5Ro4/9fvd+hgcrZm33tVuWHYGKPjmXmuz9CsVcUhfET3IGXnFejJOVGK6BGskd2DlJqdp593HtX1PYKVkZOvZ+ZGa/Gu4p7OPS+OKu5pldw+68YYZeUVamd8uu76YK36d2mrFXuPyb9FU/348ECFtPLWkt1Hde/HG12vuaNPR00cFKY73l2rv90argc+LdszeWPPEP177OWuz0pRkdF3W47o8k6tdO2ry4vfl4cH6s7317rGyHULbKm9RzNd23hzXC9N+WabsvIKXcv2v3S99h3L1MOfb9Geoxmu5d880F/HM3O1OzFDRUVGk0dcpC2HUvTcDzv1ws3h6tHBTy/8uFPtfJx6YHAXScX/F2blFmj8rPXqGtBSUyMu0d8X7laf0NaauXC3jJGC/ZppzbRry/2cvPK7y3R5x1a6MKCljmXkKvZ4lu54b62aNHLotl4d9NiIbhWOV6wNhA8A9VJRkdH//mejurRrqWnXX2LJPqfPi9bHqw/oseu6aeKgC7Rkd5Ku7tLWrcejInkFRfJq7KgycBUWGcUkZeiiQJ8yZdOy8xWTlKHeoa1r3BM37v21Wv1bsmaO6aGxV3Uqt8z2w6n6dtNhDb04QBNmbdAFbVtoyeNDXOsnf7VVc7Yc0df393cbc1ad04/P/RCtWasO6MPxfXTtJYGVlq2OnPxCPfBp8Zi2P/QLrfbrzuzFzCsoUreni0+xzr73Kg3u1s6t/LvLf1NMUqZe+d1lKjLFPazNm1Y8ymBXQrqSM/M0sGtb7T2aoX//EqNHh3fThQEttWrfcaVm5yu0TXOFt/dTYZFRlydPB77SPZR5BUXaezRDaSfza/1H2+7EdP3fkn36f6fqVVpJ+Pj0vr4a2LVufyxWhvABAKcUFRnFJmfpgrYtGtzNJXMLCrUvKVOXBvtWq+5xJ7LVzsfpNs7LGKP0nAL5eVcdts50Lq+tazFHM/TbsUyNCg+2fN9j31ujtftPaNnjQ9S5GrMn61pJ+Pjsj33rvKe6MjX5/mbAKYDzWqNGDnVp17LqgvWQs0ljdQ/xq7rgKeWNT3I4HGcdHs7ltXWta6BPhTMB69qX/9vfI/utSkgr76oL1ROEDwAAGrCv7++v5Mzcal3Dqr4gfAAA0IBVdf2o+ogbywEAAEsRPgAAgKUIHwAAwFKEDwAAYCnCBwAAsBThAwAAWIrwAQAALEX4AAAAliJ8AAAASxE+AACApQgfAADAUoQPAABgKcIHAACwVL27q60xRpKUnp7u4ZoAAIDqKvneLvker0y9Cx8ZGRmSpI4dO3q4JgAAoKYyMjLk5+dXaRmHqU5EsVBRUZHi4+Pl4+Mjh8NRq9tOT09Xx44dFRcXJ19f31rddn1n17bbtd0Sbbdj2+3absm+ba9P7TbGKCMjQyEhIWrUqPJRHfWu56NRo0bq0KFDne7D19fX4wfJU+zadru2W6Ltdmy7Xdst2bft9aXdVfV4lGDAKQAAsBThAwAAWMpW4cPpdOrZZ5+V0+n0dFUsZ9e227XdEm23Y9vt2m7Jvm1vqO2udwNOAQDA+c1WPR8AAMDzCB8AAMBShA8AAGApwgcAALCUbcLHW2+9pbCwMDVr1ky9e/fWypUrPV2lCs2YMUNXXnmlfHx8FBAQoFtuuUV79uxxKzNhwgQ5HA63R79+/dzK5Obm6uGHH1bbtm3VokUL3XTTTTp8+LBbmZSUFN19993y8/OTn5+f7r77bqWmprqVOXTokG688Ua1aNFCbdu21Z///Gfl5eXVSdunT59epl1BQUGu9cYYTZ8+XSEhIfL29taQIUMUHR3d4NstSZ07dy7TdofDoYceekjS+XPMV6xYoRtvvFEhISFyOByaO3eu2/r6doyjoqI0ePBgeXt7q3379nr++eerde+KmrY9Pz9fTzzxhHr06KEWLVooJCRE99xzj+Lj4922MWTIkDKfg7Fjxzbotkv17/NdW22vqt3l/Zt3OBx65ZVXXGUa6jGvlLGBL7/80nh5eZn333/f7Ny50zzyyCOmRYsW5uDBg56uWrlGjhxpZs2aZXbs2GG2bt1qRo8ebTp16mQyMzNdZcaPH29GjRplEhISXI/k5GS37TzwwAOmffv2JjIy0mzevNkMHTrU9OzZ0xQUFLjKjBo1yoSHh5vVq1eb1atXm/DwcHPDDTe41hcUFJjw8HAzdOhQs3nzZhMZGWlCQkLMpEmT6qTtzz77rOnevbtbu5KSklzrZ86caXx8fMy3335roqKizB133GGCg4NNenp6g263McYkJSW5tTsyMtJIMkuXLjXGnD/HfMGCBeapp54y3377rZFkvvvuO7f19ekYp6WlmcDAQDN27FgTFRVlvv32W+Pj42P+8Y9/1HrbU1NTzfDhw81XX31ldu/ebdasWWP69u1revfu7baNwYMHm4kTJ7p9DlJTU93KNLS2G1O/Pt+12faq2l26vQkJCeajjz4yDofD/Pbbb64yDfWYV8YW4eOqq64yDzzwgNuyiy++2EydOtVDNaqZpKQkI8ksX77ctWz8+PHm5ptvrvA1qampxsvLy3z55ZeuZUeOHDGNGjUyCxcuNMYYs3PnTiPJrF271lVmzZo1RpLZvXu3Mab4H06jRo3MkSNHXGW++OIL43Q6TVpaWm010eXZZ581PXv2LHddUVGRCQoKMjNnznQty8nJMX5+fuadd94xxjTcdpfnkUceMV26dDFFRUXGmPPzmJ/5n3F9O8ZvvfWW8fPzMzk5Oa4yM2bMMCEhIa7jUlttL8/69euNJLcfSoMHDzaPPPJIha9pqG2vT5/vump7dY75zTffbIYNG+a27Hw45mc670+75OXladOmTRoxYoTb8hEjRmj16tUeqlXNpKWlSZL8/f3dli9btkwBAQHq1q2bJk6cqKSkJNe6TZs2KT8/363dISEhCg8Pd7V7zZo18vPzU9++fV1l+vXrJz8/P7cy4eHhCgkJcZUZOXKkcnNztWnTptpvrKSYmBiFhIQoLCxMY8eO1f79+yVJsbGxSkxMdGuT0+nU4MGDXfVtyO0uLS8vT59++qnuvfdetxssnq/HvER9O8Zr1qzR4MGD3S7gNHLkSMXHx+vAgQO1/wacIS0tTQ6HQ61atXJb/tlnn6lt27bq3r27Hn/8cdfdwEvq3FDbXl8+35467kePHtX8+fN13333lVl3vh3zendjudp2/PhxFRYWKjAw0G15YGCgEhMTPVSr6jPGaPLkyRo4cKDCw8NdyyMiInT77bcrNDRUsbGxeuaZZzRs2DBt2rRJTqdTiYmJatq0qVq3bu22vdLtTkxMVEBAQJl9BgQEuJU5871r3bq1mjZtWifvX9++ffXJJ5+oW7duOnr0qF588UUNGDBA0dHRrv2VdywPHjzoqm9DbPeZ5s6dq9TUVE2YMMG17Hw95qXVt2OcmJiozp07l9lPybqwsLCzaWa15OTkaOrUqRo3bpzbDcPuuusuhYWFKSgoSDt27NC0adO0bds2RUZGuurVENtenz7fnjrus2fPlo+Pj8aMGeO2/Hw85ud9+ChR+tejVPylfuay+mjSpEnavn27fv31V7fld9xxh+vv8PBw9enTR6GhoZo/f36ZD25pZ7a7vPfgbMrUloiICNffPXr0UP/+/dWlSxfNnj3bNfjsbI5lfW/3mT788ENFRES4/Uo5X495eerTMS6vLhW9trbk5+dr7NixKioq0ltvveW2buLEia6/w8PD1bVrV/Xp00ebN29Wr169KqxbfW97fft8e+K4f/TRR7rrrrvUrFkzt+Xn4zE/70+7tG3bVo0bNy7ziy0pKalMCqxvHn74Yc2bN09Lly5Vhw4dKi0bHBys0NBQxcTESJKCgoKUl5enlJQUt3Kl2x0UFKSjR4+W2daxY8fcypz53qWkpCg/P9+S969Fixbq0aOHYmJiXLNeKjuW50O7Dx48qMWLF+uPf/xjpeXOx2Ne345xeWVKTgXU1XuRn5+v3//+94qNjVVkZGSVt0nv1auXvLy83D4HDbXtpXny8+2Jtq9cuVJ79uyp8t+9dH4c8/M+fDRt2lS9e/d2dU+ViIyM1IABAzxUq8oZYzRp0iTNmTNHS5YsqVZXV3JysuLi4hQcHCxJ6t27t7y8vNzanZCQoB07drja3b9/f6WlpWn9+vWuMuvWrVNaWppbmR07dighIcFV5ueff5bT6VTv3r1rpb2Vyc3N1a5duxQcHOzqdizdpry8PC1fvtxV3/Oh3bNmzVJAQIBGjx5dabnz8ZjXt2Pcv39/rVixwm064s8//6yQkJAy3dO1oSR4xMTEaPHixWrTpk2Vr4mOjlZ+fr7rc9BQ234mT36+PdH2Dz/8UL1791bPnj2rLHteHPNaHb5aT5VMtf3www/Nzp07zaOPPmpatGhhDhw44OmqletPf/qT8fPzM8uWLXObWpWdnW2MMSYjI8M89thjZvXq1SY2NtYsXbrU9O/f37Rv377MdMQOHTqYxYsXm82bN5thw4aVOy3tsssuM2vWrDFr1qwxPXr0KHd61rXXXms2b95sFi9ebDp06FBnU04fe+wxs2zZMrN//36zdu1ac8MNNxgfHx/XsZo5c6bx8/Mzc+bMMVFRUebOO+8sdxpmQ2t3icLCQtOpUyfzxBNPuC0/n455RkaG2bJli9myZYuRZF577TWzZcsW14yO+nSMU1NTTWBgoLnzzjtNVFSUmTNnjvH19T3rqYeVtT0/P9/cdNNNpkOHDmbr1q1u//Zzc3ONMcbs27fPPPfcc2bDhg0mNjbWzJ8/31x88cXmiiuuaNBtr2+f79pse1Wfd2OKp7g2b97cvP3222Ve35CPeWVsET6MMebNN980oaGhpmnTpqZXr15u01brG0nlPmbNmmWMMSY7O9uMGDHCtGvXznh5eZlOnTqZ8ePHm0OHDrlt5+TJk2bSpEnG39/feHt7mxtuuKFMmeTkZHPXXXcZHx8f4+PjY+666y6TkpLiVubgwYNm9OjRxtvb2/j7+5tJkya5TcWqTSXXdPDy8jIhISFmzJgxJjo62rW+qKjIPPvssyYoKMg4nU4zaNAgExUV1eDbXWLRokVGktmzZ4/b8vPpmC9durTcz/f48eONMfXvGG/fvt1cc801xul0mqCgIDN9+vSznnZYWdtjY2Mr/Ldfcq2XQ4cOmUGDBhl/f3/TtGlT06VLF/PnP/+5zPUwGlrb6+Pnu7baXtXn3Rhj3n33XePt7V3m2h3GNOxjXhmHMXVx6TIAAIDynfdjPgAAQP1C+AAAAJYifAAAAEsRPgAAgKUIHwAAwFKEDwAAYCnCBwAAsBThAwAAWIrwAQAALEX4AAAAliJ8AAAASxE+AACApf4/WAN9W91YkqkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(stepi, lossi);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8a1110e7",
   "metadata": {},
   "source": [
    "The plot's thickness is due to mini-batch gradient descent. More precisely, different mini-batches have a bit deviating, different loss values.<br>\n",
    "This is because the mini-batches are randomly drawn from the training set.<br>\n",
    "In total, though, they visibly converge and follow the same trend.\n",
    "\n",
    "> In general, the larger the batch size, the more stable the training process. But also the more memory is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "a76fd770",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2207682132720947\n"
     ]
    }
   ],
   "source": [
    "# Training loss\n",
    "emb = C[Xtr] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 300)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Ytr)\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b747bd00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2277145385742188\n"
     ]
    }
   ],
   "source": [
    "# Validation loss\n",
    "emb = C[Xdev] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 300)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Ydev)\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "0af7592f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.229692220687866\n"
     ]
    }
   ],
   "source": [
    "# Test loss\n",
    "emb = C[Xte] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,6) @ W1 + b1) # (32, 300)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Yte)\n",
    "print(loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "51ef9e66",
   "metadata": {},
   "source": [
    "The losses for the training set, the validation set and the test set are in the same vicinity.<br>\n",
    "This is good. It means that the model is not overfitting.<br>\n",
    "\n",
    "Let's now visualize the embeddings that were trained by the model (the weights and biases of the embedding layer).<br>\n",
    "We visualize `C`, because that is where we put the trainable, two-dimensional embedding vectors.<br>\n",
    "Let's see if the training affected these embeddings in a meaningful way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "6278bda1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqsAAAKUCAYAAAAjN3GJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABelElEQVR4nO3deXxU9b3/8ffMJIQ1gRADIYQEBJTFECUEcEFta1yqouVSubZorVb9WWuBKyjW24q3rRVtpbYutUWxLhdEVKxSC71VQUH2NCLIIkmAsMRAyIYmw8z5/YETCZkzmSSzfCd5PR8PHo/OmXNOvvl0xrznO9/FYVmWJQAAAMBAzmg3AAAAALBDWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgrLhoNyDUvF6v9u/frx49esjhcES7OQAAADiFZVmqrq5Wv3795HQG7jttd2F1//79ysjIiHYzAAAA0Iy9e/eqf//+Ac8Ja1h96KGH9Nprr+nTTz9Vly5ddO655+rhhx/WGWecEfC6999/XzNmzNAnn3yifv36adasWbr99tuD+pk9evSQdOKXT0xMbPPvEGput1vLly9Xfn6+4uPjo90cY1AXe9TGHrXxj7rYozb2qI1/1MVeW2pTVVWljIyMhtwWSFjD6vvvv68f//jHGjNmjI4fP66f/exnys/P19atW9WtWze/1xQVFemKK67Qj370I7344ov68MMPdccdd+i0007TpEmTmv2Zvq/+ExMTjQ2rXbt2VWJiIi/6k1AXe9TGHrXxj7rYozb2qI1/1MVeKGoTzJDNsIbVd955p9Hj5557Tqmpqdq4caMmTJjg95qnn35aAwYM0Lx58yRJw4YN04YNG/Too48GFVYBAADQfkR0zGplZaUkKTk52facNWvWKD8/v9GxSy+9VPPnz5fb7W6S3Ovq6lRXV9fwuKqqStKJtO92u0PV9JDxtcnEtkUTdbFHbexRG/+oiz1qY4/a+Edd7LWlNi25xmFZltXin9AKlmVp4sSJqqio0KpVq2zPGzp0qH7wgx/ovvvuazi2evVqnXfeedq/f7/S0tIanf/AAw9ozpw5Te7z8ssvq2vXrqH7BQAAABASx44d0/XXX6/Kyspmh21GrGf1zjvvVGFhoT744INmzz11/IIvT/sb1zB79mzNmDGj4bFvwG5+fr6xY1ZXrFihSy65hLEvJ6Eu9qiNPWrjH3WxR23sURv/qIu9ttTG9014MCISVn/yk5/ozTff1MqVK5tdnqBv3746ePBgo2NlZWWKi4tT7969m5yfkJCghISEJsfj4+ONflGZ3r5ooS72qI09auMfdbFHbexRG/+oi73W1KYl54d1ByvLsnTnnXfqtdde07/+9S8NHDiw2WvGjx+vFStWNDq2fPly5ebm8iIBAADoYMIaVn/84x/rxRdf1Msvv6wePXro4MGDOnjwoL744ouGc2bPnq0bbrih4fHtt9+ukpISzZgxQ9u2bdOzzz6r+fPn6+677w5nUwEAAGCgsIbVp556SpWVlbrooouUlpbW8G/RokUN5xw4cEB79uxpeDxw4EAtW7ZM7733nnJycvQ///M/evzxx1m2CgAAoAMK65jVYBYaWLBgQZNjF154oTZt2hSGFgEAACCWhLVnFQAAAGgLwioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMQwr7f51ZdiWUS2WwUAAEBobCmt1OINe7Wu+Ih2ldXI7bEU73JocGp35WUla3JuhkamJ0W7mSFDWAUAAIgBxeW1mrWkUOuKjsjldMhzUo+q22Np24Fq7ThUo+fXlChvYLLmTspWVkq3KLY4NBgGAAAAYLilBaXKf2ylNpZUSFKjoHoy3/GNJRXKf2yllhaURqyN4ULPKgAAgMGWFpRq2sICtWRkqsdrySNL0xYWSJIm5qSHpW2RQM8qAACAoYrKazVzcWGLgurJLEkzFxequLw2lM2KKMIqAACAoe5ZUiiP1bbZ/h7L0qwlhSFqUeQRVgEAAAz08b5KrSs6Yjs+NVger6V1RUe0pbQyRC2LLMasAgAAGOjVjXsV53TouJ+w2q2TS7+69izlj+ijmi+P608rd+uS4X20dX+VHnxra5PzXU6HFm/YG5NLWhFWAQAADLSu+IjfoCpJ9185XLlZvXTL8xtUXlOnGZecoRH9ErV1f5Xf8z1eS+uLK8LZ3LBhGAAAAICBdpXV+D3erZNLk87pr1+9vU2rPzusHYdqNHPxv+VyOgLeb2dZdTiaGXaEVQAAAMN4vZbcHv+9qgN6d1WnOKf+vfdow7HquuPa/XngGf9ujxWTW7MSVgEAAAzjdDoU7/LfU+rQieOnxk5H4I5VxbsccjbT+2oiwioAAICBBqd293u85HCt6o97NSqjZ8Ox7glxyuodeGvVIak9Qtm8iGGCFQAAgIHyspK141BNk6Wraus9WrJpn+67fJgqj7lVXlOn6ZcMldeyZNlsH+ByOjQmq1ckmh1y9KwCAAAYaHJuhu0aq798a6s27anQ/B/k6qVbxmpjSYU+K6tRndvr93yP19Lk3IxwNjds6FkFAAAw0Mj0JOUNTNbGkgq/vavTFhU0PO4S79JPvzlEL6/b2+Q+LqdDozN7xeQaqxI9qwAAAMaaOylbLj8zp0b0S9TVo/ppQHJXjeiXqN9PyZEkrdh6sMm5LodDcydlh7upYUPPKgAAgKGyUrrpkcnZmrawoMlo1B9dMEiDTusmt8erj0srNfnpNao45m50jkPSI5OzlZUSePKVyQirAAAABpuYky5Jmrm4UB7Lksdr6ZP9Vbrqjx/YXuNyOuRyOPTI5OyG62MVwwAAAAAMNzEnXcunT9DozBMz+u12q/Idz83speXTJ8R8UJXoWQUAAIgJWSnd9Mpt47WltFKLN+zV+uIK7SyrlttjKd7l0JDUHhqT1UuTczNidjKVP4RVAACAGDIyPalRGPV6rZjcmSpYDAMAAACIYe05qEqEVQAAABiMsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUwnNdrRbsJAABETVy0GwCgsS2llVq8Ya/WFR/RrrIaOeXV3Dxp0lOrdXZmb03OzdDI9KRoNxMAgIggrAKGKC6v1awlhVpXdEQup0Oer3pUE1wnnt9+qFqfHKzV82tKlDcwWXMnZSsrpVsUWwwAQPgxDAAwwNKCUuU/tlIbSyokqSGonsp3fGNJhfIfW6mlBaURayMAANFAzyoQZUsLSjVtYYFaMjLV47XkkaVpCwskSRNz0sPSNgAAoo2eVSCKisprNXNxYYuC6sksSTMXF6q4vDaUzQIAwBiEVSCK7llSKI/Vttn+HsvSrCWFIWoRAABmIawCUfLxvkqtKzpiOz41WB6vpXVFR7SltDJELQMAwByMWQWi5NWNexXndOi4n7DaJd6lX147UpeN6Kva+uMqK9mlv56Vpk/2V+nBt7Y2Od/ldGjxhr0saQUAaHfoWQWiZF3xEb9BVZLuu2KYxg/qrdte2Kibn1urlJSUgEHU47W0vrgiXE0FACBqCKtAlOwqq/F7vGsnl747pr9+vWybPthVrh2HqrVp0yY5HY6A99tZVh2OZgIAEFWEVSAKvF5Lbo//XtXM3l2VEOfSppKve0rdbreKyv2H24ZzPBZbswIA2h3CKhAFTqdD8S7/PaUOBe5BtRPvcsjpbN21AACYirAKRMng1O5+jxcfrlX9ca/OHtCr4Vh8fHyzW6sOSe0R0vYBAGACVgMAoiQvK1k7DtU0WbrqWL1Hr2zYq9lXnKmKY/WqOlans88eqkDLsbqcDo3J6mV/AgAAMYqwCkTJ5NwMPb+mxO9zv162TV07ufSXG3NVW3dcZXs+054vOtney+O1NDk3I1xNBQAgahgGAETJyPQk5Q1MlsvPONNj9R7NeOXfGv7zf+j83/xTu3btsr2Py+lQ3sBk1lgFALRLhFUgiuZOyparmSWpmuNyODR3UnaIWgQAgFkIq0AUZaV00yOTs1s5/19ySHpkcnazk68AAIhVjFkFomxiTrokaebiQnksq8mEK58b5q9RnedErHU5HXI5HHpkcnbD9QAAtEf0rAIGmJiTruXTJ2h05okZ/f7GsZ58PDezl5ZPn0BQBQC0e/SsAobISummV24bry2llVq8Ya/WF1d8tYXqiZ7WM/skKiczWZNzM5hMBQDoMAirgGFGpic1CqN1dfV6552/69X/N17x8fFRbBkAAJHHMADAcGyhCgDoyAirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYK6xhdeXKlbrqqqvUr18/ORwOvfHGGwHPf++99+RwOJr8+/TTT8PZTAAAABgqLpw3r62t1ahRo3TTTTdp0qRJQV+3fft2JSYmNjw+7bTTwtE8AAAAGC6sYfXyyy/X5Zdf3uLrUlNT1bNnz9A3CAAAADElrGG1tc4++2x9+eWXGj58uO6//35dfPHFtufW1dWprq6u4XFVVZUkye12y+12h72tLeVrk4ltiybqYo/a2KM2/lEXe9TGHrXxj7rYa0ttWnKNw7Isq8U/oRUcDodef/11XXPNNbbnbN++XStXrtTo0aNVV1enF154QU8//bTee+89TZgwwe81DzzwgObMmdPk+Msvv6yuXbuGqvkAAAAIkWPHjun6669XZWVlo6Gf/hgVVv256qqr5HA49Oabb/p93l/PakZGhsrLy5v95aPB7XZrxYoVuuSSSxQfHx/t5hiDutijNvaojX/UxR61sUdt/KMu9tpSm6qqKqWkpAQVVo0cBnCycePG6cUXX7R9PiEhQQkJCU2Ox8fHG/2iMr190UJd7FEbe9TGP+pij9rYozb+URd7ralNS843fp3VzZs3Ky0tLdrNAAAAQBSEtWe1pqZGu3btanhcVFSkgoICJScna8CAAZo9e7ZKS0v117/+VZI0b948ZWVlacSIEaqvr9eLL76oJUuWaMmSJeFsJgAAAAwV1rC6YcOGRjP5Z8yYIUm68cYbtWDBAh04cEB79uxpeL6+vl533323SktL1aVLF40YMUJvv/22rrjiinA2EwAAAIYKa1i96KKLFGj+1oIFCxo9njVrlmbNmhXOJgEAACCGGD9mFQAAAB0XYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIwV1rC6cuVKXXXVVerXr58cDofeeOONZq95//33NXr0aHXu3FmDBg3S008/Hc4mAgAAwGBhDau1tbUaNWqU/vjHPwZ1flFRka644gpdcMEF2rx5s+677z7dddddWrJkSTibCQAAAEPFhfPml19+uS6//PKgz3/66ac1YMAAzZs3T5I0bNgwbdiwQY8++qgmTZoUplYCAADAVGENqy21Zs0a5efnNzp26aWXav78+XK73YqPj29yTV1dnerq6hoeV1VVSZLcbrfcbnd4G9wKvjaZ2LZooi72qI09auMfdbFHbexRG/+oi7221KYl1zgsy7Ja/BNaweFw6PXXX9c111xje87QoUP1gx/8QPfdd1/DsdWrV+u8887T/v37lZaW1uSaBx54QHPmzGly/OWXX1bXrl1D0nYAAACEzrFjx3T99dersrJSiYmJAc81qmdVOhFqT+bL0qce95k9e7ZmzJjR8LiqqkoZGRnKz89v9pePBrfbrRUrVuiSSy7x21PcUVEXe9TGHrXxj7rYozb2qI1/1MVeW2rj+yY8GEaF1b59++rgwYONjpWVlSkuLk69e/f2e01CQoISEhKaHI+Pjzf6RWV6+6KFutijNvaojX/UxR61sUdt/KMu9lpTm5acb9Q6q+PHj9eKFSsaHVu+fLlyc3N5gQAAAHRAYQ2rNTU1KigoUEFBgaQTS1MVFBRoz549kk58hX/DDTc0nH/77berpKREM2bM0LZt2/Tss89q/vz5uvvuu8PZTAAAABgqrMMANmzYoIsvvrjhsW9s6Y033qgFCxbowIEDDcFVkgYOHKhly5Zp+vTpeuKJJ9SvXz89/vjjLFsFAADQQYU1rF500UUKtNjAggULmhy78MILtWnTpjC2CgAAALHCqDGrAAAAwMkIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKdDBerxXtJgAAELS4aDcAQHhtKa3U4g17ta74iHaV1cjtsRTvcmhwanflZSVrcm6GRqYnRbuZAAD4RVgF2qni8lrNWlKodUVH5HI65DmpR9XtsbTtQLV2HKrR82tKlDcwWXMnZSsrpVsUWwwAQFMMAwDaoaUFpcp/bKU2llRIUqOgejLf8Y0lFcp/bKWWFpRGrI0AAASDnlWgnVlaUKppCwvUkpGpHq8ljyxNW1ggSZqYkx6WtgEt5fVacjod0W4GgCgirALtSFF5rWYuLmxRUD2ZJWnm4kKN6t+TIQGICsZYAzgVYTVE+PQPE9yzpFAeq22z/T2WpVlLCvXKbeND1CqgeYyxBmCHsNpKfPqHaT7eV6l1RUdadE28yyG3p3G49XgtrSs6oi2llbyGERFLC0o1c/HXH7SCHWP9yORshqwAHQBhtYX49A9Tvbpxr+KcDh0PsI7qwlvHafvBark9Xn3nnP7aeaha1z3zUZPzXE6HFm/YS1hF2DHGGkBzWA2gBZhhDZOtKz4SMKj6TBrdX8e9lv7jqdW67/WP/Z7j8VpaX1wR6iYCjTQ3xvrRydl6Zupo2+t9Y6yLy2vD0j4AZqBnNUiBPv0vvHWctu6v0oNvbW10nE//iKRdZTVBnVdyuFa/+funzZ63s6y6rU0CAmpujPWcN7fK0cxUAMZYA+0fPatBCNUMaz79I1y8XqvJ2FM7hfsqgzrP7bHYmhVh4xtjbfcNlSRV1x1X1ZfHA97n5DHWQEfXXv+bTc9qEEI5w/qlH+aGqFXA15xOh9/JUv58Ue8J6p7xLgcrXCBsghlj/ejkbCV2jtetL2wMeC/GWKOj6iiTvQmrzWjNDGt/fJ/+tx2oCkGrgKYGp3bXtgOh++p+SGqPkN0LOFWwY6yDwRhrdDQdbbI3wwCa4fv0Hwoup0Ovb2ayFcIjLytZrhC+Vsdk9QrJvQB/gh1jHSzGWKOj6IiTvQmrzQj1p/9NJUdDci/gVJNzMwKO/2sJj9fS5NyMkNwLOFVLxlgHizHW6Ah8k73rPd6g/3vv8Vqq93g1bWFBzAZWhgE0I9Sf/j/7vFrKDOktAUnSyPQk5Q1M1saSCtv/iE3xs6bqqVxOh0Zn9moX45xgppaMsQ4WY6zR3nXk7bTpWQ0gLJ/++eSPMJo7KVuu5tb6aYbL4dDcSdkhahHg3+DU7iG9H2Os0d6FcrJ3rCGsBuD79B9K8XzyRxhlpXTTI5Oz1dpXmUPSI5NjeyA+YgNjrDsWhmi0TaCl3q7PG6CPZn+zyZrEf74hV7+dPKrRsVhd6o1hAM0I9Qzr00/rIYlZqwgf3+YTvr3WgxnX5HI65HI42GsdETM5N0PPrykJyb0YY22ejrKkUqQEWurt7Y8P6BdXD9f4Qb21+rPDkqTELnGaMDRFtzy/ocn5sbjUG2G1GXlZydpxqCYkE1dcTofOyewpwirCbWJOukb172m7tImP73huZi89HONLmyC2BDPGupPLqdpm1gVmjLVZOtqSSpESaLJ35RdurdzxuSbmpDeE1W+flabKY259uKu8yfmxuNQbwwCaEcwM6ynPfNRkq1V/PF5L155NrxUiIyulm165bbze+sn5+v7YARqeltgwrCXe5dDwtER9f+wAvfWT87XotvH8wUDE2Y2xdjlP9MCdk9lLOw8F/maLMdbm6IhLKkVKc5O939i8X5eP7KtOrhOx7pqcdP2tcL/s4kusLfVGz2ozgvn0Hwzfp/9haYkq2hzCBgLNGJme1KjXyeu1mDUNI/jGWE9bWNBohvMZfXpoyf87V2t2H9aLa+2HCjDG2hy+JZVa8lfS47XkkaVpCwskiSFINoKZ7P3PbYf0G8dZuvjMVBXuO6oxWcn6n7ftO9F8S73Fyt8CwmoQ5k7KVv5jK+Vp9YIRfPqHOWLlP07oGPyNsd56oErDfv6O7TWMsTZLR15SKRKCWeqt7rhX//jkoK45u5+yendVUXmttpTa75gZa0u9MQwgCMywBoDwmZiTruXTJ2h05okZ/XarBPiO52b20vLpEwiqhujISypFSjBLvb2xeb++cUaqvpubodebGVoRa0u90bMaJGZYA0D4+MZY+2aRry+u0M6y6oZZ5ENSe2hMVi9mkRvGt6RSW528pBL//zYVzGTv1Z+V6+gXbp2e2j3gOOBYXOqNsNoCzLAGgPBijHVsOXVJpYW3jtOnB6vl9VqaNLq/6o979bsV2/XG5v16cOIIXX5Wmg7X1OkXSz/Rezs+b3SvWFxSKVKCWerNa0ljf/1/zd4rFpd6I6y2EJ/+ASByCKpm87ek0qRz0vWnlbs18Y8f6MpR/fTLa85S/vC++scnB/XEu7t08/mD9LvrcnTub/5PX7q9DdfF4pJKkRLqyd6xlk8Iq63Ep38AQEfnb0mlbQeq9cd/7ZIkPfnuLv2/C0/XkWP1Wrh+ryTp8f/bqanjMzWsb6I27z3a6NpYW1IpkjryZG8mWIUIQRUA0JHYLan06cGvZ6F7LaniWL22H/w6hH5eUydJ6t29U5NrfUsqoamOPNmbnlUAANBidksqHfcTYI97vE2OOf1sCBFrSypFWked7E3PKgAAaJVgllRqiVhbUikaOuJSb/SsAgCAVglmSaVgxeKSStHS0SZ7E1YBAECrBLOkUrBicUmlaOsok70JqwAAoFVOXVJpyjMfNTnn/IffbXIs6963Gz2O1SWVTNMeg6rEmFUAANAGcydly+VnslRLxOqSSogMwioAAGi1jrykEiKDYQAAAKBNOuqSSogMelYBAECbdcQllRAZ9KwCAICQ6GhLKiEyCKsAACCkOsqSSogMhgEAAICwIqiiLQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFQAAAMYirAIAAMBYhFUAAELM67Wi3QSg3YiLdgMAAIh1W0ortXjDXq0rPqJdZTVyeyzFuxwanNpdeVnJmpyboZHpSdFuJhCTCKsAALRScXmtZi0p1LqiI3I5HfKc1KPq9ljadqBaOw7V6Pk1JcobmKy5k7KVldItii0GYg/DAAAAaIWlBaXKf2ylNpZUSFKjoHoy3/GNJRXKf2yllhaURqyNQHtAzyoAAC20tKBU0xYWqCUjUz1eSx5ZmrawQJI0MSc9LG0D2puw96w++eSTGjhwoDp37qzRo0dr1apVtue+9957cjgcTf59+umn4W4mAABBKSqv1czFhU2C6sJbx+nnVw5v9npL0szFhSourw1L+4D2JqxhddGiRZo2bZp+9rOfafPmzbrgggt0+eWXa8+ePQGv2759uw4cONDwb8iQIeFsJgAAQbtnSaE8Vttm+3ssS7OWFIaoRUD7FtZhAL/73e90880365ZbbpEkzZs3T//4xz/01FNP6aGHHrK9LjU1VT179gzqZ9TV1amurq7hcVVVlSTJ7XbL7Xa3vvFh4muTiW2LJupij9rYozb+URd7ba3N1v1V+veew4pzSHGuxs85JLkclhJcwQRZS//ec1iFew5rWFpiq9oSarxu/KMu9tpSm5Zc47CsNn48tFFfX6+uXbtq8eLFuvbaaxuO//SnP1VBQYHef//9Jte89957uvjii5WVlaUvv/xSw4cP1/3336+LL77Y9uc88MADmjNnTpPjL7/8srp27RqaXwYAgGacd955qqqqksfjUWZmprxer4qLi7V9+/ZoNw0wzrFjx3T99dersrJSiYmBP7CFrWe1vLxcHo9Hffr0aXS8T58+OnjwoN9r0tLS9Mwzz2j06NGqq6vTCy+8oG9+85t67733NGHCBL/XzJ49WzNmzGh4XFVVpYyMDOXn5zf7y0eD2+3WihUrdMkllyg+Pj7azTEGdbFHbexRG/+oi7221mbSU6u1/VC13+f+epZDw/tl6LkPd+utv32onIye+s2kHD28+qhWf1bu95oz+yTq1f83vsXtCAdeN/5RF3ttqY3vm/BghH01AIfD0eixZVlNjvmcccYZOuOMMxoejx8/Xnv37tWjjz5qG1YTEhKUkJDQ5Hh8fLzRLyrT2xct1MUetbFHbfyjLvZaW5tth2rl9vj/G2ZJ2nagWr9bsUuStKPsmK4fN1B5g1L07o7Dfq/ZeqjGth1eryWn0//PCideN/5RF3utqU1Lzg9bWE1JSZHL5WrSi1pWVtaktzWQcePG6cUXXwx18wAAaBGv15LbE3jk3KcHG/cWfV79pXp3b9qh4uP2WA2hlF2wAP/CFlY7deqk0aNHa8WKFY3GrK5YsUITJ04M+j6bN29WWlpaOJoIAEDQnE6H4l2OgIH1+CnPWZYUqHM03uXQniPH2AULCCCswwBmzJihqVOnKjc3V+PHj9czzzyjPXv26Pbbb5d0YrxpaWmp/vrXv0o6sVpAVlaWRowYofr6er344otasmSJlixZEs5mAgAQlMGp3bXtgP8xq62R2qOz8h9b2bAUVrC7YD0yOZtNBdBhhDWsXnfddTp8+LAefPBBHThwQCNHjtSyZcuUmZkpSTpw4ECjNVfr6+t19913q7S0VF26dNGIESP09ttv64orrghnMwEACEpeVrJ2HKqxDZUt4XBIpUe/aNE17IKFjijsE6zuuOMO3XHHHX6fW7BgQaPHs2bN0qxZs8LdJAAAWmVyboaeX1MSknu1ZeFI3y5Yo/r3ZEgA2r2wh1UAANqLkelJyhuYrI0lFU16V6c881GT8299YWPY2uLbBeuV28xY+goIl7ButwoAQHszd1K2XDZLMEaSx2tpXdERbSmtjHZTgLCiZxUAgBbISummRyZna9rCArX2m3ynQ/I37LV/ry764J5vNDn+0e7DfntuXU6HFm/Yy5JWaNcIqwAAtJBvYtPMxYXyWFZQE65cTodcDodSenTS/qNf+j1n/9EvNOaX/2x4fFqPBL14y1itLTri93yP19L64opW/AZA7GAYAAAArTAxJ13Lp0/Q6Mxekk6EUX98x3Mze2n59An6vLrO9p5eS/q8pk6f19Sp6ku3fnXtSG3aU6F5/9xhe83OstAtpQWYiJ5VAABaKSulm165bXzD7lPriyu0s6y6YfepIak9NCarV8PuU8HsguXz8KRsdUuI0/f/sjbgygEn74IFtEeEVQAA2mhkelKjcaN24TGYXbAk6c5vDNaFQ0/TxCc+VG29J+C58S4HQRXtGsMAAAAIsUDhcXBq94DXXjayr+76xhD9+OVN2nPkWLM/a0hqjxa3D4glhFUAACIoLyvZdnzr0D7d9bvvjtLT73+mnYdqdFr3BJ3WPUFJXeL9nu9yOjQmq1eT494Q7LAFmIJhAAAARFCgXbCy+/dU105xuuubQ3TXN4c0HLdbusrjtTQ5N6NhzOy64iPaVVbTMGZ2cGp35WUlN4yZBWIRYRUAWojJLGiLQLtgvbpxn17duC+o+7icDo3ol6gH39qqdUVH5HI6Gt3P7bG07UC1dhyq0fNrSpQ3MFlzJ2WzPStiDmEVAJpBrxVCbe6kbOU/tlKeVm8rIDkkbdtfJe9Xj+3WevUd31hSofzHVuqRydkN68QCsYCwCgA2istrNWtJIb1WCLlQ7IJ1PIhxqQtvHaet+6v04Ftb5fFa8sjStIUFkkRgRcxgghUA+LG0oFT5j63UxpITuwMF22u1tKA0Ym1EbJuYk655U3LUyeW0nXB1KtdXS1+5HK0fhmLpxM5bxeW1rb4HEEmEVQA4xdKCUk1bWKB6jzeobTSlE6G13uPVtIUFBFYErTW7YA1LSzwxBqANPJalWUsK23aTAFiNAKHEMAAAOElRea1mLi5s9Vezvl6rUf17MiQAQWnJLliWJV31xw/83qdLvEu/vHakLhvRV7V1x/XMqt22P9PjtbSu6Ii2lFaGZLw147oRToRVADjJPUsK5Qm0t2UQfL1Wr9w2PkStQkcQzC5Yv1i6RXFOh9/xqvddMUzjB/XWbS9s1OfVdZp52RkamZ6krfur/P48l9OhxRv2tilEMq4bkcAwAAD4ysf7KrWu6EjQX/3bObnXCmgtf8ujrSs+4jeodu3k0nfH9Nevl23TB7vKtf1Qtf7rlX8HHNvq8VpaX1zR6vYxrhuRQs8qAHzl1Y17bXutFt46TtsOVKnuuFdTxmTI7fHqpbV7NO+fO/3eKxS9VsCpdpXV+D2e2burEuJc2lTydfis/MKt3eX+z/fZWVbdqnb4xnW35GMdqxGgtehZBYCv2PVa+Uwa3V9f1Ht0zRMf6qG/f6q7vjFE5w9O8XtuW3utgFN5vZbcHv+vT0crZ1y5PVaLJ0MFM67719eepYKfX6Li33xbw9MSGz3HagTmMnViHD2rAPAVu14rn08PVOv3/3eiJ7X48DHdMD5L5w3urQ92lfs9v7W9VoA/zq+WrfIXWIsP16r+uFdnD+il/R8fkCQldonTwJRuWrv7iO09412OFu/G1ty47ouGnqb/GN1fU575SHuPHNORY/VNzmFctxliZWIcYRUAFLjXyufTg40nqnxe/aV6d0+wPd/Xa8XWrAiVwandte1A0w9Bx+o9emXDXs2+4kxVHKtXeU2dZl56hprrKBuS2qNFP983rjuQAb27qqz6S23aY//NQqhXI0DLxNrEOMIqAChwr5XP8VOesywpUA5tTa8VEEheVrJ2HKrxO5np18u2qWsnl/5yY65q647rz6uK1KNzvO29XE6HxmT1atHPDzSuW5IenZyt/xidIUkq/s23ta/imM5/+F3bn8+47shbWlCqmYu/7h2PhW16CasA8BW7XqvWammvFdqfUPesT87N0PNrSvw+d6zeoxmv/FszXvl3w7FnVgZea3VybkaLfn5z47rnvLlVJYeP6T/zBmjiHz8MOFyAcd2R529i3Mlb8vpjwsQ4wioAfCVQr1VLtabXCrEv3GMAR6YnKW9gsjaWVLTpdepyOjQ6s1eL29LcuO7quuOqrTsur2Xp85q6Zu/HuO7IieUNT1gNAAC+Mjk3IyRBVWpdrxViV3F5rb77pzW68g8f6MW1e7TtQHXDkBLfGMAX1+7RlX/4QN/905o2zYSfOyk74PqpwXA5HJo7KbtF1wQzrrulWrMaAVonlBueRBo9qwDwlUC9VlOe+ajJ+be+sNHvfVrba4XYFOkxgFkp3fTI5OwWr3Pq45D0yOSWT5gJZlx3SzGuOzKamxjncjo05+oRuvbsdHm8ll5cW6LfLt/R5LxoTYyjZxUAThKtXivEJt8YwHqPN+heeY/XUr3Hq2kLC1q9m9PEnHTNm5KjTi6nXEGGPZfToU4up+ZNyWn1uMPBqd1bdZ0dxnVHhm9inJ1Jo/vL47V0zRMf6oG/faKbzx+oKWP8fzPkmxgXSYRVADiJr9eqtXG1tb1WiD2hGgPY2iEBE3PStXz6BI3OPDE22i60+o7nZvbS8ukT2jRBJi8rOehw3BzGdUdOcxPjDhz9Qg++tVW7y2u1tGC/nl9drJvPH+j33GhMjCOsAsApotVrhdhiwhjArJRueuW28XrrJ+fr+2MHaHhaouJdJ16z8S6Hhqcl6vtjB+itn5yvRbeNb/OHKMZ1x6bmJsZt3nu00eNNe44qK6Wb7dJ8kZ4Yx5hVAPBjYk66RvXvabtwto/veG5mLz0c5YWzETnBLI4fjFCNARyZntTo+nBtRhHMagTPflisZz8sDngfxnVHTnuYGEfPKgDYiHSvFWKHvzGA3xyWqsJf5Ms35Hl4WqKKf/Ntzb78zIZzfn3tSD0+JafRdeEYAxjOSUuM644tvolxgZyd0bPJ4+LyWtsd0CI9MY6eVQBoRqR6rRA7/I0BXLf7iLolxGlEv0RtKa3S2EHJOlxTp7GDejecM3ZQbz37QVGj62JtcfxorUaA1mtuw5O0nl10/7eH6eW1ezQyPUk3npulX729zfb8SE+Mo2cVAFqIoAp/YwCr645r6/4qjfsqnI4b1FvzPyjSsLQe6tbJpdO6J+j007rro92Hm1wba4vjM647tjQ3Me61TfvUOd6lN+48Tw9OHKHnVxfr5XV7/J4bjYlx9KwCANACgcYAflR0WOMG9dZfVhVpTFayHv3Hdl0+Mk1jspKV2CVen1d/qc8+bzr73zcG0LQPQoHaxLju2BFom96T15C+/40tzd4rGhPjCKsAALRAoMXxP9p9WNflZmh4WqK8lqWdZTVaW3RYYwclK6lLvNbu9j8py6TF8X+9bJs+Kj4a1HaxvnHdvm1m1xdXaGdZdcN1Q1J7aExWrzZvM4u2ifY2vW1FWAUAoIXsxgD6xq3+8PyshmC6tuiI7rjodCV2iddzNrPko704fnF5re57rUDX9ZEWbdirY+6vn/NtF7vjUI2eX1OivIHJmntKDynjus03d1K28h9bKU+rVwaO3sQ4xqwCANBCdmMAfeNWr8lJbxiburbosEb0S7IdrxrtxfGXFpQq/7GVKvhqrc1gt4sNtPsWQdU8sbzhCWEVAIAWCrQ4/prdhxXncjYE06ovjmtXWbXKa+r8TsyK5uL40douFtERqxPjGAYAAEALBRoD+Otl2/TrZY2X/bni8Q/83ieai+PbbRf715vH65P9VXrwra0Br/dtFzuqf08mTcWQWJwYR88qACDmRXI3HZ9YXxzfhO1iER2xtuEJPasAgJjjm32+rvhIULPWwyGWF8c3bbtYREesTIwjrAIAYkZxea3t15fBzFoPNd8YvpmLT/RSBjPu0+V0yOVw6JHJ2VEbA+jbLvbUXbj8uXDoafrD9WfrgTc/0Wubmo5R9W0XS1iNfSYGVYlhAACAGOGbtb6x5MTWpKGYtR4KE3PStXz6BI3OPDGj327iiu94bmYvLZ8+Iaq7OPnbLtafq7LT9Mfrz9Z/vfJvv0FVir3tYhF76FkFABjPN2u9JV+3e7yWPLI0bWGBJIU1HMba4vj+ViU41ffHZWrWpWfo1r9u1Bo/S26dLNa2i0VsIawCAIxmN2s9WCfPWk9P6hTKpjURC2MAA20X63PZyL5K6Z6gyU+v1r/3VTZ7T1O3i0X7wDAAAIDRYnnWuonhzbddbCBbD1TpSG190Ou/mrRdLNofwioAwFi+Wett2c9c+nrW+rYDVSFqWWwbnNo94PN7Dh/Tf/75I10yvI/mXD2i2ftFe7tYtG+EVQCAsXyz1v1J7tZJ63/2Td1x0ekNx3IyemrHLy/XBUNSmpzvcjr0+mZ2XJLst4s9WVF5rf7zmY90+ci++vmVw23Pi/Z2sWj/CKsAAGMFmrV+pLZeM18t1LRvDdVZ6Unq2smlx67L0YsflWjVzvIm53u8ljaVHA1zi2NDoO1iT7a7vFb/+ee1umpUP/3s28P8nhPN7WLRMTDBCgBgrOZmrb+3/XMtXL9H86bkqHBfpeqOe/TwO5/anv/Z59VSZqhbGXsCbRd7w/w1qvN83ev62ec1GvOrf/q9TzS3i0XHQc8qAMBIwcxal6Rfvb1NcU6Hvn1WmqYtLFDdca/tue4obMtqqljfLra1orE1L9qGnlUAgJF8s9abC6wDkruqT2JnOR1Seq8u+vSg/Zqf8TE4Yz1cS0LF8naxLWHC1rxoG8IqAMBYg1O7a9uBAOHT5dDvp+TorcL9+qysVg9PytZl81aqvKbe7/mnn9ZDktm7LUUyXJ28XazLad8jfTITtosNhmlb86L1CKsAAGPlZSVrx6Ea28lAd+efoR6d4/XAm1tVW39cF51xmh6elK2bn9/Q5FyX06FzMnvK1LAarXA1MSddo/r31H2vFUgqP7FKgKfpeb425Wb20sOGB7ulBaWaufjr9XmD3ZrX9ADeUTFmFQBgrECz1scNStYPzx+o6YsKVFN3XJYlTV9UoDFZyfr+2AFNzvd4LV17tplBZGlBqfIfW6mNJSeCdLDhamlBaJbiykrppudvypMkXZeboeFpiQ0bB8S7HBqelqjvjx2gt35yvhbdNt74oDptYYHqPd6g1+f1eC3Ve7yatrAgZDVF6NCzCgAwVqBZ6x/tPqIhP/t7o2P7K79U9pzlTe7jm7U+LC1RRZvD2uQW84Wrlowb9XgteWRp2sICSQppb+B9VwxTfHy8JDO3iw0klFvzmhzIOxp6VgEARmvPs9ZDFa6Ky2tD2awGsRRUpdjemhf2CKsAAKP5Zq23NjaZPGs9ULhaeOu4gDtH+RCuTgj11rxbSitD1DK0FcMAAADGO3nWuseyggokps9a94Wrtjo5XHXkJZh8W/P62/Fs4a3jtP2rJc2uPTtdHq+lF9eW6LfLd/i9l8vp0OINezt0PU1CzyoAICZMzEnX8ukTNDrzxD70dnvb+47nZvbS8ukTjAyq0tfhyp9HJ2dr3KDe+uH5A1X8m2+r+DffVv9eXWzv5QtXHVmgrXkladLo/vJ4LV3zxId64G+f6ObzB2rKGP/bxHq8ltYXm7lqREdEzypiWqwN/gfQNlkp3fTKbeMb1iJdX1yhnWXVDWuRDkntoTFZvWJiofdA4WrOm1s1MKW7th+s1mMrTvT+Ha6ts70X4ar5rXkPHP1CD761VZK0u7xWZ/btoZvPH6iF6/2H/J1l9uv7IrIIq4gp7EQCQDqxSsDJ7/VY/OAaKFxV1x2X2+PVl26PPq+xD6kn68jhKpiteTfvPdro8aY9R3XLBYPkdEj+PjO4PRZbsxqCsIqYwE4kAAIJVVCNVOgNJly1lC9cxVpoD4Vgt+ZtiXiXQ06nQx4/GyQgsgirMB47kQAIl2h9WxPOcNVRNbc179kZPZs8Li6v9durKklDUnuEsHVoC8IqjGbaYtkA2oc9h49p9tKtUf22prlwVX/c26Lw2dHDVXNb86b17KL7vz1ML6/do5HpSbrx3Cz96u1tfs91OR0ak9UrnM1FC7AaAIxl+mLZAGLXNU98GLWtTX3yspJtVzSQpH0VXygno6f69+qiXl3jFWhfBMJV4K15Jem1TfvUOd6lN+48Tw9OHKHnVxfr5XV7/J7r8VqanOt/pQBEHmEVxmInEgChtuzjA5Kkem/0941vLlz9edVueb2WVky/UJt/nq/0nvZLVxGuvt6a1+4DwHGPpfvf2KLsB5Yr58EVmvuP7X7PczkdyhuYzGRdgxBWYSR2IgEQakXltbr/9S2tvj7U39Y0F66Kymv1nadWa9jP31HWvW9rX8UXfs8jXH2tPW/N25ERVmEku8Wyv3NOujb/9yXq5Gr80n3q++fot98d5fdeLJYNQPrq25pWDyw6IdTf1hCuQqs9b83bkRFWYSS7xbLfLjwgl9Ohbw1PbTjWq2u8vnFmql7dsM/vvVgsG4Cp39YQrkJvYk665k3JUSeXs6HXesozHzVsCOCPy+lQJ5dT86bkMCnXQKwGACPZLZZdd9yrpQX7NXl0hpZ9fFCSdM3Z6TpY+aXW7D5se7+OvFg2gJO/rfEfVi8f2Vc//dYQZfXupi/qPfpkf5V+9NcN+sLddJHNUO8b7wtHviX6ggnULqdDLoeDJfpsTMxJ16j+PW3X5/bxHc/N7KWHWZ/bWIRVGKe5xbIXrt+jpT8+T30SE3Soqk6TR/fXqxv996r6dOTFsgF8/W2Ny9X0udN6JOjx/zxbv/n7p/rHJwfVrVOcxgxMtp19H45vawhXodeetubt6AirME5zi2V/sr9K2w5Ua9I5/fX+js91Rt9E3fz8hoD37OiLZQMdXaCtTVN7JCje5dQ7Ww6q9OiJSUzbDwX+NiYc39YQrsKjPWzN29ERVmGk5hbLXrR+j354/kD1SeysD3eV60DllwHv19EXywY6sua+rdl2oEof7CzXO9Mu0Mod5Vq183Mt23JAVV8ct70mnN/WEK7Ci1rGHiZYwUjNLZb9RsF+9U3qrCl5GXqlmZn+LJYNdGy+b2vseC3p+/PX6gfPrdeusmrdeG6W/vVfF6l/L/t1TSP5bQ3hCh0dYRVGam6x7Jq64/r7loM6VufR8k8OBbwXi2UDGJzavdlzNpZU6LF/7tS3H18lt8erS0f0tT2Xb2uAyCGswkjNLZYtnRhn9kZBqeo9XttzWCwbgBT425qcjJ6646LTdVZ6kvolddZlI/squVsnfWYzzpVva4DIIqzCWHaLZSd1iddV2Wk69/QUvbCmJOA9WCwbgBT425rqL49r7MBkPXfTGL1790X6r/wz9Ku3t+m9HZ/7PZ9va4DIYoIVjOVbLHvawoJGKyO+fdf5SuwSr9/8/VPtDrDtIYtlA/DxfVuzZd+RJs999nmNbnxufVD3cTkdGp3Zi29rgAgirMJo/hbLPv/hdwNew2LZAPyZOylbV/3+/Tbdg29rzMJKCR0DYRXGY7FsAKGQldJNv7x2pLR3c6uu59ua6POtQbuu+Ih2ldU0rEE7OLW78rKSWYO2nSKsIiawWDaAULjirDQt27tZnZxOHbfE1qYxori81rbDwu2xtO1AtXYcqtHza0qUNzBZc+mwaFcIq4gpLJYNIBTe+PF5mr10K9/WxIClBaUNQ8Ek+w8YvuMbSyqU/9hKPly0I4RVxDSCKoDWGNC7K9/WxIClBaVNJtk2x+O15JGlaQsLJInA2g4QVgEAHRbf1pirqLxWMxcXtiionszSicm5o/r3pFc8xrHOKgAAXyGomuOeJV9/9d9aHsvSrCWFIWoRooWeVQAAYJSP91VqXVHTNXF9HA7ptgmDNGXMAKX17Kzymnq9vHaPnnh3V6PzPF5L64qOaEtpJcM5YhhhFQAAGOXVjXsV53TouM1kqnsuPVNT8jL0P29t1friCqX2SNDpqd39nutyOrR4w17CagwjrAIAAKOsKz5iG1S7dXLppvOy9PM3P9GSTaWSpD1HjmlDSYXf8z1eS+uL/T+H2MCYVQAAYJRdZTW2zw1O7a6EeJc+3FUe9P12llWHolmIEsIqAAAwhtdrye2xn1j1pdvb4nu6PZa8QWwAATMRVgEAgDGcTofiXfarMhQfrtUX9R6dNzgl6HvGuxys9BDDGLMKAACMMji1u7Yd8P/Vfd1xr55+/zPNvvxMuT1ebSiuUO9unTSkTw+9smGv32uGpPYIZ3MRZoRVAABglLysZO04VGO7terj/9qp415LMy4ZqtQenVVW/aVeXrvH77kup0NjsnqFs7kIM8IqAAAwyuTcDD2/psT2ecuSnnh3V5N1Vf3xeC1Nzs0IZfMQYYxZBQAARhmZnqS8gclytXGcqcvpUN7AZNZYjXGEVQAAYJy5k7LlcrQxrDocmjspO0QtQrQQVgEAgHGyUrrpkcnZam1cdUh6ZHK2slK6hbJZiALGrAIAACNNzEmXJM1cXCiPZdlOuDqZy+mQy+HQI5OzG65HbKNnFQAAGGtiTrqWT5+g0ZknZvTbjWP1Hc/N7KXl0ycQVNsRelYBAIDRslK66ZXbxmtLaaUWb9ir9cUV2llWLbfHUrzLoSGpPTQmq5cm52YwmaodIqwCAICYMDI9qVEY9XotdqbqABgGAAAAYhJBtWMgrAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxwh5Wn3zySQ0cOFCdO3fW6NGjtWrVqoDnv//++xo9erQ6d+6sQYMG6emnnw53EwEAAGCosIbVRYsWadq0afrZz36mzZs364ILLtDll1+uPXv2+D2/qKhIV1xxhS644AJt3rxZ9913n+666y4tWbIknM0EAACAocIaVn/3u9/p5ptv1i233KJhw4Zp3rx5ysjI0FNPPeX3/KeffloDBgzQvHnzNGzYMN1yyy364Q9/qEcffTSczQQAAIChwrbdan19vTZu3Kh777230fH8/HytXr3a7zVr1qxRfn5+o2OXXnqp5s+fL7fbrfj4+CbX1NXVqa6uruFxVVWVJMntdsvtdrf11wg5X5tMbFs0URd71MYetfGPutijNvaojX/UxV5batOSa8IWVsvLy+XxeNSnT59Gx/v06aODBw/6vebgwYN+zz9+/LjKy8uVlpbW5JqHHnpIc+bMaXJ8+fLl6tq1axt+g/BasWJFtJtgJOpij9rYozb+URd71MYetfGPuthrTW2OHTsW9LlhC6s+DkfjfXsty2pyrLnz/R33mT17tmbMmNHwuKqqShkZGcrPz1diYmJrmx02brdbK1as0CWXXOK3p7ijoi72qI09auMfdbFHbexRG/+oi7221Mb3TXgwwhZWU1JS5HK5mvSilpWVNek99enbt6/f8+Pi4tS7d2+/1yQkJCghIaHJ8fj4eKNfVKa3L1qoiz1qY4/a+Edd7FEbe9TGP+pirzW1acn5YZtg1alTJ40ePbpJ1/CKFSt07rnn+r1m/PjxTc5fvny5cnNzeYEAAAB0QGFdDWDGjBn6y1/+omeffVbbtm3T9OnTtWfPHt1+++2STnyFf8MNNzScf/vtt6ukpEQzZszQtm3b9Oyzz2r+/Pm6++67w9lMAAAAGCqsY1avu+46HT58WA8++KAOHDigkSNHatmyZcrMzJQkHThwoNGaqwMHDtSyZcs0ffp0PfHEE+rXr58ef/xxTZo0KZzNBAAAgKHCPsHqjjvu0B133OH3uQULFjQ5duGFF2rTpk1hbhUAAABiQdi3WwUAAABai7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVbbGa/XinYTAAAAQiYu2g1A22wprdTiDXu1rviIdpXVyO2xFO9yaHBqd+VlJWtyboZGpidFu5kAAACtQliNUcXltZq1pFDrio7I5XTIc1KPqttjaduBau04VKPn15Qob2Cy5k7KVlZKtyi2GAAAoOUYBhCDlhaUKv+xldpYUiFJjYLqyXzHN5ZUKP+xlVpaUBqxNgIAAIQCYTXGLC0o1bSFBar3eJuE1IW3jtPPrxze5BqP11K9x6tpCwsIrAAAIKYwDCCGFJXXaubiQtlNobrthY067vHaXm9Jmrm4UKP692RIAAAAiAn0rMaQe5YUymPZz/av/MKt2npPwHt4LEuzlhSGumkAAABhQViNER/vq9S6oiO241Ml+2EAJ/N4La0rOqItpZWhbiIAAEDIEVZjxKsb9yrO6QjJvVxOhxZv2BuSewEAAIQTYTVGrCs+ouMhWvDf47W0vrgiJPcCAAAIJ8JqjNhVVhPS++0sqw7p/RB+7E4GAOiIWA0gBni9ltye0AYVt8eS12vJGaKhBQg93+5km0sO6+ZMKfvB5fLKye5kAIAOhbAaA5xOh+JdjpAG1niXg6BqqFN3J4tzeKXME89Fc3cyPtwAAKKBsBojBqd217YDofvqfkhqj5DdC6GztKBUMxd/vUSZx2spztX0vFN3J3tkcrYm5qSHtC2+nt11xUe0q6xGbo+leJeDnl0AQEQRVmNEXlaydhyqCbh0VbBcTofGZPUKQasQSr7dyVry/7DHa8kjS9MWFkhSSALrqT27J7/motmzCwDomJhgFSMm52Y0G1SnPPORHnxra7P38ngtTc7NCFXTEALN7U7WHN/uZMXltW1qx9KCUuU/tlIbS06sFmH3mju1Z5dtfAEA4UJYjREj05OUNzBZrjaOGXQ5HcobmMzXt4ZpbneyYLR1dzJfz269xxt0D77Ha6ne49W0hQUEVgBAWBBWY8jcSdlyOdoYVh0OzZ2UHaIWIRSC2Z0sGG3ZncyUnl0AAE5FWI0hWSnd9MjkbLU2rjokPTKZ8YWmCbQ72Qf3XKwbzx3Y6Niyu87XtG8N8Xt+a3cnM6FnFwAAfwirMWZiTrrmTclRJ5cz6CEBLqdDnVxOzZuSE/IZ42i7aO9OZkLPLgAAdlgNIAZNzEnXqP49bWds+/iO52b20sPM2DZWtHcn8/Xs2gXmTi6nZl9xpq4a1U89EuJUWFqp/3lrqwr3NQ2lvp5dxkQDAEKFsBqjslK66ZXbxjeshbm+uEI7y6ob1sIcktpDY7J6sRam4UzYnay5nt3ZV5ypy0em6e5X/q19R7/Q7RcO0l9/mKcLH3lPlV+4G53bmp5dAAACIazGuJHpSY3CKLsMxZbmdifzepsei3MFHr3T0t3JAvXsdol36XtjM3X34n/rvR2fS5LuXfKxPrjnNF03JkPPrNzd5JqW9uwCABAIY1bbGYJq7Bmc2t32uSO1dUrt0bnhcfeEOGX06hrwfi3Znay5nt3M3l3VKc7ZsO6qJB33Wvr3vqO27fb17AIAEAqEVSDK8rLs189d/dlhXX12upKTkzUktYd++91RAWftt3R3Ml/Prh3fSmnWKT/TIcmuGS3t2QUAIBDCKhBlgXYne/K9z7Sh6IjGjRunZ27M0/JPDmrPYfu1TFuzO1mgnt3i8mOqO+5RblZyw7E4p0Nn9U+yHT7Qkp5dAACaw5hVIMp8u5NtLKloElpr6o5r+qJNmpvn0ax1LtV5HFqyyf9OUS6nQ6Mze7V4Ql1eVrJ2HKrxG5i/cHv00kd7dN8Vw1T5hVulX02w6hLv0qINe/y2oSU9uwAANIeeVcAA0dydLFDPriQ9/M6n+vuWA/rdd0fp7Z+cr8ze3XTDs+tU9cXxJue2pmcXAIBA6FkFDODbnWzawoJWbXnalt3JAvXsSlLdca/m/G2r5vxta8D7tLZnFwCAQOhZBQwRzd3JotmzCwBAIIRVwCATc9K1fPoEjc48Me7TLrT6judm9tLy6RPavI2ur2e3tXG1LT27AAAEwjAAwDCn7k5WUHJE0ol1TsO5O5kv8M5cXCiPZQUcx+rjcjrkcjj0yOTsNgdmAAD8IawChvLtTuZ2u7Vs2TIV/jxfCQmdwvozJ+aka1T/npq1pFDrio7I5XT4Da2+47mZvfTwJHpUAQDhQ1gFYkSkFto/tWd3fXGFdpZVy+2xwtqzCwCAP4RVAH75enZ9vF6LnakAABHHBCsAQSGoAgCigbAKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWIRVAAAAGIuwCgAAAGMRVgEAAGAswioAAACMRVgFAACAsQirAAAAMBZhFUC74vVa0W4CEDN4vyAWxEW7AQDQVr9etk0fFR/VrrIauT2W4l0ODU7trrysZE3OzdDI9KRoNxEwwpbSSi3esFfrio80eb+My+qpnGg3EPCDsAogJhWX1+q+1wp0XR9p0Ya9Oub++jm3x9K2A9XacahGz68pUd7AZM2dlK2slG7RazAQRcXltZq1pFDrio7I5XTIc1KPqu/9UlJerZxc6cbn1unX38nh/QJjMAwAQMxZWlCq/MdWqmDvUUlq9If3ZL7jG0sqlP/YSi0tKI1UEwFj+N4vG0sqJDX/finYe5T3C4xCzyqAmLK0oFTTFhbIkuRQcOPtPF5LHlmatrBAkjQxJz18DQQMcvL7JVger6V6j5f3C4xBzyqAmFFUXquZiwtb9If3ZJakmYsLVVxeG8pmAUbi/YL2grAKIGbcs6RQHqtts5c9lqVZSwpD1CLAXLxf0F4QVgHEhI/3VWpd0RHb8XbB8ngtrSs6oi2llSFqGWAe3i9oTxizCiAmvLpxr+KcDh3388f3giGn6baLhuiMPj3k8VratKdCc/62VXuOHPN7L5fTocUb9rKkFdqtQO+XluL9gmijZxVATFhXfMT2D2+XTi79ZVWRrv7jB/reX9bKa0l/mjpaDof/e3m8ltYXV4SxtUB0BXq/tBTvF0QbPasAYsKushrb55Z/clB1nq+T6T1LCrXpvy/RkNTu2nHI/3U7y6pD3kbAFIHeL63B+wXRRFhFzPF6LTmdNl1maJe8Xktuj30vUUZyV/3km2fo7Ixe6tUtXs6vulT79exiG1bdHovXEtql5t4vrcH7BdFEWIXxAm0PyHaaHYPT6VC8y2H7B/jpqWO0/+iXuve1Qh2qqpPTIa2YcaE6uexHOsW7HPzhRbvU3PulNXi/IJoIqzBWMNsDsp1mxzE4tbu2HWj6VWR8fLwGp/bQ7Nc+bhhXl5vZq9n7DUntEfI2Aqawe7+0Fu8XRBMTrGCklm4PyHaa7V9eVrJcfnp23G63Kmrr9Z95A5TZu6vGn95b9185POC9XE6HxmQ1H2iBWGX3fjnZDeMz9dItY5u9F+8XRBthFcbxbQ9Y7/EGvUbgydsDEljbp8m5Gbavh+mLNums9CQtnzZBP79yuB5ati3gvTxeS5NzM8LRTMAIgd4vPsndOimzd9dm78X7BdHGMAAYpeTwsZBsDziqf0+GBLQzI9OTlDcwWRtLKpr8EV7zWbkueWxlo2NZ977t9z4up0OjM3sxzhntWqD3i8+8f+7UvH/uDHgf3i8wAT2rMMrP39zC9oCwNXdStlx2i6cGyeVwaO6k7BC1CDAX7xe0F4RVGCVQL0Cw2B6w/cpK6aZHJmertX9+HZIemcxEPHQMvF/QXjAMAEaJczpU5/H/3G0TBul7YzOVmpigovJaPf5/O/X3LQf9nsv2gO3XxJx0SSeGe7ic3qCucTkdcjkcemRydsP1QEdw8vvFY1lBdQa4nA51kpP3C4xBWIVRTmwP2LQf4O78M3TZyL66/42PVXS4VmMH9ta863J0pHad1hYdaXI+2wO2bxNz0jWqf0/d91qBpPITs579fMjxLXmWm9lLD7O0GToo3/vFbilAH9/qAWdn9NSvvpPD+wXGIKzCeF3iXbrlgoG6/s8fadOeo5KkvUf2KTerl64fO8BvWJXYHrC9y0rppudvytOyZct0XW6G1hZXamdZdcOmEUNSe2hMVi82jQB04v3yym3jGzZZWV9c0eT9MjYrSVKRFtyUp/j4+Gg3GWhAWIURvAG+mhrSp7s6x7v0ws2N1wOMdzm1db/9uFS2B+w47rtiWMMfV/4/B+yNTE9q9OHt5PeL2+3WsmVF0WoaYIuwCiMEChe+fd5/uGC9DlZ92ei5+uP2YxbZHrBj4v9zIHi8XxALCKsw3s5D1apze9SvZxfbr/z9YXtAAABiH2EVRvE3Uaa23qNnVu3Wf185XE6HtL64Qj06x+mcAb10rP64lmxqumMV2wMCANA+EFZhFI/NagC/Xb5Dh2vqdcdFg5WR3FVVX7r1SWmlnnjvM9v7sD0gAACxj7AKo4zO7KW1xZV+l1VZsLpYC1YXN3sPtgcEAKD9YAcrGOV/rh7J9oAAAKABYRVGGdC7K9sDAgCABgwDgHFauz0g22kCAND+0LMKI03MSdfy6RM0OvPEjH6XzVqAvuO5mb20fPoEgioAAO0MPaswVjDbA7KdJgAA7RthFcYLtD0gAABo3xgGgJhDUAUAoOMgrAIAAMBYhFUAAAAYi7AKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmEVAAAAxiKsAgAAwFhhDasVFRWaOnWqkpKSlJSUpKlTp+ro0aMBr/nBD34gh8PR6N+4cePC2UwAAAAYKi6cN7/++uu1b98+vfPOO5KkW2+9VVOnTtXf/va3gNdddtlleu655xoed+rUKZzNBAAAgKHCFla3bdumd955Rx999JHGjh0rSfrzn/+s8ePHa/v27TrjjDNsr01ISFDfvn3D1TQAAADEiLCF1TVr1igpKakhqErSuHHjlJSUpNWrVwcMq++9955SU1PVs2dPXXjhhfrVr36l1NRUv+fW1dWprq6u4XFVVZUkye12y+12h+i3CR1fm0xsWzRRF3vUxh618Y+62KM29qiNf9TFXltq05JrHJZlWS3+CUH49a9/rQULFmjHjh2Njg8dOlQ33XSTZs+e7fe6RYsWqXv37srMzFRRUZH++7//W8ePH9fGjRuVkJDQ5PwHHnhAc+bMaXL85ZdfVteuXUPzywAAACBkjh07puuvv16VlZVKTEwMeG6Le1btwuHJ1q9fL0lyOBxNnrMsy+9xn+uuu67hf48cOVK5ubnKzMzU22+/re985ztNzp89e7ZmzJjR8LiyslIDBgzQ+PHj1aNHj2Z/n0hzu9169913dfHFFys+Pj7azTEGdbFHbexRG/+oiz1qY4/a+Edd7LWlNtXV1ZJO5MLmtDis3nnnnZoyZUrAc7KyslRYWKhDhw41ee7zzz9Xnz59gv55aWlpyszM1M6dO/0+n5CQ0KjH1TcMYODAgUH/DAAAAERedXW1kpKSAp7T4rCakpKilJSUZs8bP368KisrtW7dOuXl5UmS1q5dq8rKSp177rlB/7zDhw9r7969SktLC+r8fv36ae/everRo0fAHtxoqaqqUkZGhvbu3dtst3dHQl3sURt71MY/6mKP2tijNv5RF3ttqY1lWaqurla/fv2aPTdsE6yGDRumyy67TD/60Y/0pz/9SdKJpauuvPLKRpOrzjzzTD300EO69tprVVNTowceeECTJk1SWlqaiouLdd999yklJUXXXnttUD/X6XSqf//+YfmdQikxMZEXvR/UxR61sUdt/KMu9qiNPWrjH3Wx19raNNej6hPWTQFeeuklnXXWWcrPz1d+fr6ys7P1wgsvNDpn+/btqqyslCS5XC59/PHHmjhxooYOHaobb7xRQ4cO1Zo1a4wcfwoAAIDwCuumAMnJyXrxxRcDnnPywNouXbroH//4RzibBAAAgBgS1p5VNJWQkKBf/OIXfpfh6sioiz1qY4/a+Edd7FEbe9TGP+piL1K1Cds6qwAAAEBb0bMKAAAAYxFWAQAAYCzCKgAAAIxFWAUAAICxCKsAAAAwFmE1zCoqKjR16lQlJSUpKSlJU6dO1dGjRwNeU1NTozvvvFP9+/dXly5dNGzYMD311FORaXAEtaY2krRt2zZdffXVSkpKUo8ePTRu3Djt2bMn/A2OoNbWxue2226Tw+HQvHnzwtbGaGhpXdxut+655x6dddZZ6tatm/r166cbbrhB+/fvj1yjw+TJJ5/UwIED1blzZ40ePVqrVq0KeP7777+v0aNHq3Pnzho0aJCefvrpCLU08lpSm9dee02XXHKJTjvtNCUmJmr8+PHtdr3vlr5mfD788EPFxcUpJycnvA2MopbWpq6uTj/72c+UmZmphIQEnX766Xr22Wcj1NrIamltXnrpJY0aNUpdu3ZVWlqabrrpJh0+fLhtjbAQVpdddpk1cuRIa/Xq1dbq1autkSNHWldeeWXAa2655Rbr9NNPt959912rqKjI+tOf/mS5XC7rjTfeiFCrI6M1tdm1a5eVnJxszZw509q0aZP12WefWW+99ZZ16NChCLU6MlpTG5/XX3/dGjVqlNWvXz/rscceC29DI6yldTl69Kj1rW99y1q0aJH16aefWmvWrLHGjh1rjR49OoKtDr2FCxda8fHx1p///Gdr69at1k9/+lOrW7duVklJid/zd+/ebXXt2tX66U9/am3dutX685//bMXHx1uvvvpqhFsefi2tzU9/+lPr4YcfttatW2ft2LHDmj17thUfH29t2rQpwi0Pr5bWxefo0aPWoEGDrPz8fGvUqFGRaWyEtaY2V199tTV27FhrxYoVVlFRkbV27Vrrww8/jGCrI6OltVm1apXldDqt3//+99bu3butVatWWSNGjLCuueaaNrWDsBpGW7dutSRZH330UcOxNWvWWJKsTz/91Pa6ESNGWA8++GCjY+ecc451//33h62tkdba2lx33XXW97///Ug0MWpaWxvLsqx9+/ZZ6enp1pYtW6zMzMx2FVbbUpeTrVu3zpLU7B9pk+Xl5Vm33357o2Nnnnmmde+99/o9f9asWdaZZ57Z6Nhtt91mjRs3LmxtjJaW1saf4cOHW3PmzAl106KqtXW57rrrrPvvv9/6xS9+0W7Daktr8/e//91KSkqyDh8+HInmRVVLa/PII49YgwYNanTs8ccft/r379+mdjAMIIzWrFmjpKQkjR07tuHYuHHjlJSUpNWrV9ted/755+vNN99UaWmpLMvSu+++qx07dujSSy+NRLMjojW18Xq9evvttzV06FBdeumlSk1N1dixY/XGG29EqNWR0drXjdfr1dSpUzVz5kyNGDEiEk2NqNbW5VSVlZVyOBzq2bNnGFoZfvX19dq4caPy8/MbHc/Pz7etw5o1a5qcf+mll2rDhg1yu91ha2uktaY2p/J6vaqurlZycnI4mhgVra3Lc889p88++0y/+MUvwt3EqGlNbd58803l5uZq7ty5Sk9P19ChQ3X33Xfriy++iESTI6Y1tTn33HO1b98+LVu2TJZl6dChQ3r11Vf17W9/u01tIayG0cGDB5WamtrkeGpqqg4ePGh73eOPP67hw4erf//+6tSpky677DI9+eSTOv/888PZ3IhqTW3KyspUU1Oj3/zmN7rsssu0fPlyXXvttfrOd76j999/P9xNjpjWvm4efvhhxcXF6a677gpn86KmtXU52Zdffql7771X119/vRITE0PdxIgoLy+Xx+NRnz59Gh3v06ePbR0OHjzo9/zjx4+rvLw8bG2NtNbU5lS//e1vVVtbq+9+97vhaGJUtKYuO3fu1L333quXXnpJcXFxkWhmVLSmNrt379YHH3ygLVu26PXXX9e8efP06quv6sc//nEkmhwxranNueeeq5deeknXXXedOnXqpL59+6pnz576wx/+0Ka2EFZb4YEHHpDD4Qj4b8OGDZIkh8PR5HrLsvwe93n88cf10Ucf6c0339TGjRv129/+VnfccYf++c9/hu13CpVw1sbr9UqSJk6cqOnTpysnJ0f33nuvrrzyypiYLBLO2mzcuFG///3vtWDBgoCvLROF+/3k43a7NWXKFHm9Xj355JMh/z0i7dTfubk6+Dvf3/H2oKW18fnf//1fPfDAA1q0aJHfD0axLti6eDweXX/99ZozZ46GDh0aqeZFVUteM16vVw6HQy+99JLy8vJ0xRVX6He/+50WLFjQ7npXpZbVZuvWrbrrrrv085//XBs3btQ777yjoqIi3X777W1qQ/v9uBRGd955p6ZMmRLwnKysLBUWFurQoUNNnvv888+bfFLx+eKLL3Tffffp9ddfb+g2z87OVkFBgR599FF961vfavsvEEbhrE1KSori4uI0fPjwRseHDRumDz74oPWNjpBw1mbVqlUqKyvTgAEDGo55PB7913/9l+bNm6fi4uI2tT2cwlkXH7fbre9+97sqKirSv/71r5jtVZVOvA9cLleTno2ysjLbOvTt29fv+XFxcerdu3fY2hppramNz6JFi3TzzTdr8eLFxv93tqVaWpfq6mpt2LBBmzdv1p133inpRECzLEtxcXFavny5vvGNb0Sk7eHWmtdMWlqa0tPTlZSU1HBs2LBhsixL+/bt05AhQ8La5khpTW0eeughnXfeeZo5c6akE/mlW7duuuCCC/TLX/5SaWlprWoLYbUVUlJSlJKS0ux548ePV2VlpdatW6e8vDxJ0tq1a1VZWalzzz3X7zVut1tut1tOZ+NOb5fL1dCzaLJw1qZTp04aM2aMtm/f3uj4jh07lJmZ2fbGh1k4azN16tQmf2AvvfRSTZ06VTfddFPbGx9G4ayL9HVQ3blzp959992YD2edOnXS6NGjtWLFCl177bUNx1esWKGJEyf6vWb8+PH629/+1ujY8uXLlZubq/j4+LC2N5JaUxvpRI/qD3/4Q/3v//5vm8fWmaildUlMTNTHH3/c6NiTTz6pf/3rX3r11Vc1cODAsLc5UlrzmjnvvPO0ePFi1dTUqHv37pJO/B1yOp3q379/RNodCa2pzbFjx5oMG3G5XJK+/janVdo0PQvNuuyyy6zs7GxrzZo11po1a6yzzjqryVI7Z5xxhvXaa681PL7wwgutESNGWO+++661e/du67nnnrM6d+5sPfnkk5Fufli1pjavvfaaFR8fbz3zzDPWzp07rT/84Q+Wy+WyVq1aFenmh1VranOq9rYagGW1vC5ut9u6+uqrrf79+1sFBQXWgQMHGv7V1dVF41cICd9yMvPnz7e2bt1qTZs2zerWrZtVXFxsWZZl3XvvvdbUqVMbzvctXTV9+nRr69at1vz589v90lXB1ubll1+24uLirCeeeKLR6+Po0aPR+hXCoqV1OVV7Xg2gpbWprq62+vfvb/3Hf/yH9cknn1jvv/++NWTIEOuWW26J1q8QNi2tzXPPPWfFxcVZTz75pPXZZ59ZH3zwgZWbm2vl5eW1qR2E1TA7fPiw9b3vfc/q0aOH1aNHD+t73/ueVVFR0egcSdZzzz3X8PjAgQPWD37wA6tfv35W586drTPOOMP67W9/a3m93sg2PsxaUxvLsqz58+dbgwcPtjp37myNGjWq3a0/a1mtr83J2mNYbWldioqKLEl+/7377rsRb38oPfHEE1ZmZqbVqVMn65xzzrHef//9huduvPFG68ILL2x0/nvvvWedffbZVqdOnaysrCzrqaeeinCLI6cltbnwwgv9vj5uvPHGyDc8zFr6mjlZew6rltXy2mzbts361re+ZXXp0sXq37+/NWPGDOvYsWMRbnVktLQ2jz/+uDV8+HCrS5cuVlpamvW9733P2rdvX5va4LCstvTLAgAAAOHDagAAAAAwFmEVAAAAxiKsAgAAwFiEVQAAABiLsAoAAABjEVYBAABgLMIqAAAAjEVYBQAAgLEIqwAAADAWYRUAAADGIqwCAADAWP8fZQiQ6eZSdCMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This code plots the 2-dimensional embeddings of the 27 characters\n",
    "plt.figure(figsize=(8,8))\n",
    "plt.scatter(C[:,0].data, C[:,1].data, s=200)\n",
    "for i in range(C.shape[0]):\n",
    "    plt.text(C[i,0].item(), C[i,1].item(), itos[i], ha=\"center\", va=\"center\", color=\"white\")\n",
    "plt.grid('minor')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "16afef5d",
   "metadata": {},
   "source": [
    "We can see the vowels are clustered together. This means that the model has learned that vowels are, to some extend, interchangeable and therefore somewhat alike in character.<br>\n",
    "`q` seems to be very unique, just like our special `.` character. We can see structure, it's not just pure randomness.<br>\n",
    "\n",
    "\n",
    "Let's optimize further beyond increasing the model's hidden layer.<br>\n",
    "Let'stackle the next possible bottleneck: **The dimensions of the character embeddings.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3095fe66",
   "metadata": {},
   "source": [
    "## Remove the Embedding Bottleneck\n",
    "\n",
    "For a first run, let's increase the embedding dimensions to $10$.<br>\n",
    "With the below code, the goal was to beat Andrej's validation loss of $2.17$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6f6a9aa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tqdm\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "39c9bcc4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set:\n",
      "X: torch.Size([182580, 3]) \tY: torch.Size([182580])\n",
      "Validation Set:\n",
      "X: torch.Size([22767, 3]) \tY: torch.Size([22767])\n",
      "Test Set:\n",
      "X: torch.Size([22799, 3]) \tY: torch.Size([22799])\n"
     ]
    }
   ],
   "source": [
    "block_size = 3\n",
    "\n",
    "def build_dataset(words):\n",
    "    X, Y = [], []\n",
    "    \n",
    "    for w in words:\n",
    "        context = [0] * block_size\n",
    "        for ch in w + '.':\n",
    "            ix = stoi[ch]\n",
    "            X.append(context)\n",
    "            Y.append(ix)\n",
    "            context = context[1:] + [ix] # crop and append\n",
    "    \n",
    "    X = torch.tensor(X)\n",
    "    Y = torch.tensor(Y)\n",
    "    print('X:', X.shape, '\\tY:', Y.shape)\n",
    "    return X,Y\n",
    "\n",
    "random.seed(42)          # for reproducibility\n",
    "random.shuffle(words)    # words is just the bare list of all names, from wayyy above\n",
    "n1 = int(0.8*len(words)) # index at 80% of all words (rounded for integer indexing)\n",
    "n2 = int(0.9*len(words)) # index at 90% of all words (rounded for integer indexing)\n",
    "\n",
    "print('Training Set:')\n",
    "Xtr, Ytr = build_dataset(words[:n1])     # The first 80% of all words\n",
    "print('Validation Set:')\n",
    "Xdev, Ydev = build_dataset(words[n1:n2]) # The 10% from 80% to 90% of all words\n",
    "print('Test Set:')\n",
    "Xte, Yte = build_dataset(words[n2:])     # The 10% from 90% to 100% of all words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "aa0d268e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11897 parameters\n"
     ]
    }
   ],
   "source": [
    "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
    "C = torch.randn((27, 10), generator=g)\n",
    "W1 = torch.randn((30,200), generator=g)\n",
    "b1 = torch.randn((200), generator=g)\n",
    "W2 = torch.randn((200,27), generator=g)\n",
    "b2 = torch.randn((27), generator=g)\n",
    "\n",
    "parameters = [C, W1, b1, W2, b2] # Cluster all parameters into one structure\n",
    "\n",
    "print(sum(p.nelement() for p in parameters), 'parameters')\n",
    "\n",
    "for p in parameters:\n",
    "    p.requires_grad = True\n",
    "    \n",
    "lossi = []\n",
    "stepi = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "1f79062f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 300000/300000 [08:27<00:00, 591.58it/s]\n"
     ]
    }
   ],
   "source": [
    "for i in tqdm.tqdm(range(300000)):\n",
    "    # mini-batch construct\n",
    "    ix = torch.randint(0, Xtr.shape[0], (128,)) # Batch size was 32 before, now 128\n",
    "    \n",
    "    # Forward-Pass\n",
    "    emb = C[Xtr[ix]] # (32, 3, 10)\n",
    "    h1 = torch.tanh(emb.view(-1, 30) @ W1 + b1) # (32, 300)\n",
    "    logits = h1 @ W2 + b2 # (32, 50)\n",
    "    loss = F.cross_entropy(logits, Ytr[ix]) # (32,)\n",
    "    \n",
    "    # Backward-Pass\n",
    "    for p in parameters:\n",
    "        p.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    lr = 0.1 if i < 60000 else 0.05 if i < 120000 else 0.01\n",
    "    \n",
    "    for p in parameters:\n",
    "        p.data += -lr * p.grad\n",
    "    \n",
    "    stepi.append(i)\n",
    "    lossi.append(loss.log10().item())\n",
    "    \n",
    "# print('Loss for current mini-batch:', loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "1b8fc17b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAAGdCAYAAADJ6dNTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGcUlEQVR4nO3dd1gUZ+IH8O/uAgsirCLSEbEXFBWiYseCYkkxiSYmlkQvMbFETXI/0VwsKXiXxDPNktjixahnrDmxkGgEuyIo1lhQUEAEFbDRdn5/AMvOFthFYAf3+3kengdm35l9dxjY775tZIIgCCAiIiKSCLmlK0BERESkjeGEiIiIJIXhhIiIiCSF4YSIiIgkheGEiIiIJIXhhIiIiCSF4YSIiIgkheGEiIiIJMXG0hUwhVqtRmpqKpycnCCTySxdHSIiIjKBIAjIzc2Fl5cX5HLT20NqRThJTU2Fr6+vpatBRERElZCSkgIfHx+Ty9eKcOLk5ASg+MU5OztbuDZERERkipycHPj6+mrex01VK8JJaVeOs7MzwwkREVEtY+6QDA6IJSIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSTE7nMTExGDYsGHw8vKCTCbD1q1bTd734MGDsLGxQYcOHcx9WiIiIrISZoeTBw8eIDAwEN99951Z+2VnZ2PMmDHo16+fuU9JREREVsTse+uEh4cjPDzc7Cd6++23MWrUKCgUCrNaW4iIiMi61MiYk1WrVuHKlSuYM2dOTTydyTbF3cDc7Wdx5GqWpatCREREJar9rsSXLl3CzJkzERsbCxsb054uLy8PeXl5mp9zcnKqpW77/7qN7adS0cilDro2aVAtz0FERETmqdaWk6KiIowaNQrz5s1DixYtTN4vMjISKpVK8+Xr61uNtSQiIiIpqdZwkpubixMnTmDy5MmwsbGBjY0N5s+fj1OnTsHGxgZ79+41uF9ERASys7M1XykpKdVZTSIiIpKQau3WcXZ2RmJiomjb4sWLsXfvXvz666/w9/c3uJ9SqYRSqazOqhEREZFEmR1O7t+/j8uXL2t+TkpKQkJCAlxcXNCoUSNERETg5s2bWLNmDeRyOQICAkT7u7m5wd7eXm+7JQmWrgARERFpmB1OTpw4gdDQUM3PM2bMAACMHTsWq1evRlpaGpKTk6uuhkRERGRVZIIgSL7hICcnByqVCtnZ2XB2dq6y4763Ph7bElLxj6FtML6H4S4mIiIiqpzKvn/z3jpEREQkKQwnREREJCkMJ0RERCQpDCcAasGwGyIiIqth1eFEZukKEBERkR6rDidEREQkPQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJClWHU5kMs7XISIikhqrDidEREQkPQwnREREJCkMJ0RERCQpDCcAuHo9ERGRdDCcEBERkaRYdTjhXB0iIiLpsepwQkRERNLDcEJERESSwnACQABHxBIREUkFwwkRERFJinWHE46IJSIikhzrDidEREQkOQwnREREJCkMJ0RERCQpDCfg8vVERERSwnBCREREkmLV4UTG6TpERESSY9XhhIiIiKSH4YSIiIgkheEE4OL1REREEsJwQkRERJLCcEJERESSYtXhRMbJOkRERJJj1eGEiIiIpIfhhIiIiCSF4QRcvp6IiEhKGE6IiIhIUqw6nHA8LBERkfRYdTghIiIi6WE4ISIiIklhOAEgcAF7IiIiyWA4ISIiIklhOCEiIiJJsepwwuXriYiIpMeqwwkRERFJj9nhJCYmBsOGDYOXlxdkMhm2bt1abvnNmzdjwIABaNiwIZydnRESEoLdu3dXtr5ERET0lDM7nDx48ACBgYH47rvvTCofExODAQMGICoqCnFxcQgNDcWwYcMQHx9vdmWrC5evJyIikg4bc3cIDw9HeHi4yeUXLVok+vnzzz/Htm3b8Ntvv6Fjx47mPj0RERE95cwOJ09KrVYjNzcXLi4uRsvk5eUhLy9P83NOTk611EXGBeyJiIgkp8YHxH711Vd48OABRowYYbRMZGQkVCqV5svX17cGa0hERESWVKPhZN26dZg7dy42bNgANzc3o+UiIiKQnZ2t+UpJSanBWhIREZEl1Vi3zoYNGzB+/Hhs3LgR/fv3L7esUqmEUqmsoZoRERGRlNRIy8m6deswbtw4/PLLLxgyZEhNPCURERHVUma3nNy/fx+XL1/W/JyUlISEhAS4uLigUaNGiIiIwM2bN7FmzRoAxcFkzJgx+Prrr9G1a1ekp6cDABwcHKBSqaroZRAREdHTwuyWkxMnTqBjx46aacAzZsxAx44d8fHHHwMA0tLSkJycrCm/bNkyFBYWYtKkSfD09NR8vffee1X0EiqPy9cTERFJj9ktJ3369IFQzqplq1evFv38559/mvsUREREZMV4bx0iIiKSFIYToNyWICIiIqpZDCdEREQkKQwnREREJClWHU44W4eIiEh6rDqcEBERkfQwnADgeFgiIiLpYDghIiIiSWE4ISIiIkmx8nDCEbFERERSY+XhhIiIiKSG4YSIiIgkheEEACfrEBERSQfDCREREUkKwwkRERFJilWHEy5fT0REJD1WHU6IiIhIehhOwOXriYiIpIThhIiIiCSF4YSIiIgkheGEiIiIJMWqwwkn6xAREUmPVYcTIiIikh6GEwACF7AnIiKSDIYTIiIikhSGEyIiIpIUqw4nXL6eiIhIeqw6nBAREZH0MJyAy9cTERFJCcMJERERSQrDCREREUkKwwkRERFJilWHExkXsCciIpIcqw4nREREJD0MJwAXryciIpIQhhMiIiKSFIYTIiIikhSGEyIiIpIUqw4nvLcOERGR9Fh1ONHg+vVERESSwXBCREREksJwQkRERJLCcEJERESSYtXhhONhiYiIpMeqw0kpDoclIiKSDoYTIiIikhSzw0lMTAyGDRsGLy8vyGQybN26tcJ99u/fj6CgINjb26NJkyZYunRpZepa5QrVxW0mj/KLLFwTIiIiKmV2OHnw4AECAwPx3XffmVQ+KSkJgwcPRs+ePREfH49Zs2Zh6tSp2LRpk9mVrWprjyYDAJYfSLJwTYiIiKiUjbk7hIeHIzw83OTyS5cuRaNGjbBo0SIAQOvWrXHixAl8+eWXePHFF819eiIiInrKVfuYk8OHDyMsLEy0beDAgThx4gQKCgoM7pOXl4ecnBzRFxEREVmHag8n6enpcHd3F21zd3dHYWEhMjMzDe4TGRkJlUql+fL19a3uahIREZFE1MhsHZnOHfaEknvZ6G4vFRERgezsbM1XSkpKtdeRiIiIpMHsMSfm8vDwQHp6umhbRkYGbGxs0KBBA4P7KJVKKJXK6q4aERERSVC1t5yEhIQgOjpatG3Pnj0IDg6Gra1tdT89ERER1TJmh5P79+8jISEBCQkJAIqnCickJCA5uXhabkREBMaMGaMpP3HiRFy/fh0zZszA+fPnsXLlSqxYsQIffPBB1bwCIiIieqqY3a1z4sQJhIaGan6eMWMGAGDs2LFYvXo10tLSNEEFAPz9/REVFYXp06fj+++/h5eXF7755htOIyYiIiKDZELp6FQJy8nJgUqlQnZ2NpydnavsuI1n7tB8f23BkCo7LhEREVX+/Zv31iEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIkmpVDhZvHgx/P39YW9vj6CgIMTGxpZbfu3atQgMDESdOnXg6emJN954A1lZWZWqMBERET3dzA4nGzZswLRp0zB79mzEx8ejZ8+eCA8PR3JyssHyBw4cwJgxYzB+/HicPXsWGzduxPHjxzFhwoQnrjwRERE9fcwOJwsXLsT48eMxYcIEtG7dGosWLYKvry+WLFlisPyRI0fQuHFjTJ06Ff7+/ujRowfefvttnDhx4okrX5UeFxRZugpEREQEM8NJfn4+4uLiEBYWJtoeFhaGQ4cOGdynW7duuHHjBqKioiAIAm7duoVff/0VQ4YMMfo8eXl5yMnJEX1Vt0K1UO3PQURERBUzK5xkZmaiqKgI7u7uou3u7u5IT083uE+3bt2wdu1ajBw5EnZ2dvDw8EC9evXw7bffGn2eyMhIqFQqzZevr6851awUQWA4ISIikoJKDYiVyWSinwVB0NtW6ty5c5g6dSo+/vhjxMXFYdeuXUhKSsLEiRONHj8iIgLZ2dmar5SUlMpUk4iIiGohG3MKu7q6QqFQ6LWSZGRk6LWmlIqMjET37t3x4YcfAgDat28PR0dH9OzZE59++ik8PT319lEqlVAqleZU7Ymx3YSIiEgazGo5sbOzQ1BQEKKjo0Xbo6Oj0a1bN4P7PHz4EHK5+GkUCgUAaXWlSKgqREREVs3sbp0ZM2Zg+fLlWLlyJc6fP4/p06cjOTlZ000TERGBMWPGaMoPGzYMmzdvxpIlS3D16lUcPHgQU6dORefOneHl5VV1r+RJMZwQERFJglndOgAwcuRIZGVlYf78+UhLS0NAQACioqLg5+cHAEhLSxOteTJu3Djk5ubiu+++w/vvv4969eqhb9+++Oc//1l1r4KIiIieGjJBSn0rRuTk5EClUiE7OxvOzs5VdtzGM3dovk/4eADq1bGrsmMTERFZu8q+f/PeOiWkH9GIiIisA8NJCWYTIiIiaWA4KVELereIiIisAsNJCUYTIiIiaWA4KcGGEyIiImlgOCkhsO2EiIhIEhhOSjGbEBERSQLDSYmzqTmWrgIRERGB4UTj7sN8S1eBiIiIwHCiwQGxRERE0sBwUoLZhIiISBoYTkqo2XRCREQkCQwnREREJCkMJ6XYcEJERCQJDCcluAgbERGRNDCclOCQEyIiImmw6nCitCl7+WqGEyIiIkmw6nBib6vQfM9uHSIiImmw6nAik5V9z24dIiIiabDqcKLt7gMuX09ERCQFDCcl1h1LtnQViIiICFYeTuRa/TocEEtERCQNVh1OtIacoIiDToiIiCTBqsOJdhy5nZtnsXoQERFRGasOJ0RERCQ9Vh1OZBUXISIiohpm1eGEiIiIpIfhhIiIiCTFqsOJTMaOHSIiIqmx8nBi6RoQERGRLqsOJ1zahIiISHqsOpwQERGR9Fh1OGG3DhERkfRYdTghIiIi6bHqcMKGEyIiIumx6nBCRERE0mPV4WRwO09LV4GIiIh0WHU4ebO7v6WrQERERDqsOpwoba365RMREUkS352JiIhIUqw6nHC2DhERkfRYdTghIiIi6WE4ISIiIklhONEi8E6AREREFmfd4URn0Mm2hFTL1IOIiIg0rDuc6Fh1MMnSVSAiIrJ6lQonixcvhr+/P+zt7REUFITY2Nhyy+fl5WH27Nnw8/ODUqlE06ZNsXLlykpVuCrJdJpOLt7KtVBNiIiIqJSNuTts2LAB06ZNw+LFi9G9e3csW7YM4eHhOHfuHBo1amRwnxEjRuDWrVtYsWIFmjVrhoyMDBQWFj5x5auamkNOiIiILM7scLJw4UKMHz8eEyZMAAAsWrQIu3fvxpIlSxAZGalXfteuXdi/fz+uXr0KFxcXAEDjxo2frNZVRKa70AnDCRERkcWZ1a2Tn5+PuLg4hIWFibaHhYXh0KFDBvfZvn07goOD8a9//Qve3t5o0aIFPvjgAzx69Mjo8+Tl5SEnJ0f0VRPUnK1DRERkcWa1nGRmZqKoqAju7u6i7e7u7khPTze4z9WrV3HgwAHY29tjy5YtyMzMxLvvvos7d+4YHXcSGRmJefPmmVO1KlHIfh0iIiKLq9SAWJlOf4ggCHrbSqnVashkMqxduxadO3fG4MGDsXDhQqxevdpo60lERASys7M1XykpKZWpJhEREdVCZrWcuLq6QqFQ6LWSZGRk6LWmlPL09IS3tzdUKpVmW+vWrSEIAm7cuIHmzZvr7aNUKqFUKs2pWqXw3jpERETSY1bLiZ2dHYKCghAdHS3aHh0djW7duhncp3v37khNTcX9+/c12/766y/I5XL4+PhUospERET0NDO7W2fGjBlYvnw5Vq5cifPnz2P69OlITk7GxIkTARR3yYwZM0ZTftSoUWjQoAHeeOMNnDt3DjExMfjwww/x5ptvwsHBoepeCRERET0VzJ5KPHLkSGRlZWH+/PlIS0tDQEAAoqKi4OfnBwBIS0tDcnKypnzdunURHR2NKVOmIDg4GA0aNMCIESPw6aefVt2rqCRj42SIiIjIcmRCLbjbXU5ODlQqFbKzs+Hs7Fxlx737IB8dPxF3UV1bMKTKjk9ERGTNKvv+zXvrEBERkaQwnBAREZGkMJwQERGRpFh1OOF4WCIiIumx6nBiyOOCIktXgYiIyKoxnOgoKFJbugpERERWzarDSV2l/jIvamYTIiIii7LqcGKj0H/5Px+9boGaEBERUSmrDieGfLH7oqWrQEREZNUYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhODPjp0DVLV4GIiMhqMZwYMGf7WUtXgYiIyGoxnBAREZGkMJwQERGRpDCcGCEIgqWrQEREZJUYToxYsPOCpatARERklRhOjFgWc9XSVSAiIrJKDCdEREQkKQwn5cgvVFu6CkRERFbHxtIVkLIWH+2Ea10lwgM88MnzAZauDhERkVVgy0kFMu/n4T9HruPSrVxLV4WIiMgqMJyY6HEBu3iIiIhqAsOJiQTor3tSWMTAQkREVNUYTkyk1skmcdfvovlHO7F0/xXLVIiIiOgpxXBiIt0VY19ccgiCwMXaiIiIqhpn65iotOVEEAQU6TajEBERUZVhy4mJRq84CkEQ8Nryo+gaudfS1SEiInpqMZyY6GF+ERJvZuPQlSxk3s/Te3xbwk3EXb9rgZoRERE9Xaw+nIwJ8TO57LPfHTS4PfFGNt5bn4AXlxzCrZzHnMVDRET0BKw+nChtnvwUXMt6oPm+y+d/4PUVR5/4mFLzedR5vPNznN7AYCIioqpm9eGkKkxZFy/6+cjVOwCATXE3sObwNQBAXmER3l0bh3XHkmu6elXih5ir2HkmHaduZFu6KkRE9JSz+tk6cpmsWo57Mvku3t94CgAQ1sYDv5+/hajEdEQlpuPVzo1MPs6DvELYKuSwq4IWHm2CIKBQLcBWYfi42Y8KoJDLUFcpvkQK2GVFRETVzOpbTqr6Tb/U8MWHNN9fSM/BR1vP6JUpfaO/8yAf762Px6HLmaLH7+cVou2c3ei2oOpnB72+4igC5+1B7uMCvcceFxQhcN4eBMzZrdeNY06vTtz1O0hkSwsREZnJ6ltOasI7P58U/Tx6xVE429ti55k0jO/hjx9jkwAA2xJScW3BEOQXqvEovwixl28DgGh2UPajAhy9moU+Ld2eKFgdvJwFANh5Jh0jgn1Fj92890jzfXHrivmtS/ce5uPFJYcBAEmRgyGrphYqIiJ6+lh9OKlXx67an+NRQZHo59hLZS0kpcFEW58v9iE1+7HBY41ZeQynUu4BAP5vUCu80b0x7G0V5T7/44IinE/LQaBPPcjl4pDw919Pa8LJtcwHeGP1cQxo4270WKYOiDU03ZqIiMgUVt+t81oX08d/1ISoxDSjweTwlSxNMAGAf+66gN5f7DNYtrBIjeyHBXhxySG0+scuvLD4EFYe1A9C2j7aegZJmQ/wQ8xVk+trLKzUpkk9giBgW8JNnE/LsXRViIgIDCcVtjrUtHfXnjS4/Y/zt/Dqj0f0tt/KyUPjmTuwJf4Gvtx9EY1n7sDy2Kto8dFOBM7fI1oY7tMd53H3Qb7B42fkPsYBnTEvhuQ+LsSxpDsYsfQw2s3ZjWc++wPf77uM5bFXoTayrP+tHHEryp6z6Xht+RGkGwlhNS32UibeW5+A8K9jTSr/uKAIgxbF4ONt+uOITHE/r/gcGjtfT6va9Hpv3H3Iwd8A8gvViNiciF1n0vG4oAhh/96PiM2Jlq6W5N15kI9l+68gI1ca/+NqI6sPJ7XF+J9OlPv49A2n8N2+ywCKQ4ix94GOn0Tjk/+dE2177vuD6PzZHwbL644UmbDmBEYsO4xj1+4gN68Qmffz8MXui/h0x3n8LzFNU0776btGio/91n/icPByFuZsL//NXRAEXLl93+CbWsxft8udlp11Pw/bEm7isU6XmiFnU81rMdl5Jg0X0nOx5vB1s/Yr9coPhzFi2WH850jl9n9S2Y/0B0FXt3m/nUXnz//AHSPhWEoOXclEj3/uQ58v/sSYlccQfe6WpatkMb8cvY51x5Ix8ec4RJ+7hb9u3a+1yyHUpHd+jkPkzgvo/NkfuHH3oaWrUytVKpwsXrwY/v7+sLe3R1BQEGJjTfvEefDgQdjY2KBDhw6VeVqqIisOiLt3tLuKDEnKfFDu46WuZNwvew6dsTTTNySIBtoCOgNvi9SIu34H+YVln1ZXHEhCv6/KPqmtOpiE0SuO4nFBEcasPIaIzYmiliHtGzK+vOww3lufgFb/2IVJa0+We7NGQStKmTKmRv2EH6jP3CwOQ5tO3jBapqBIXS0hYuWBJATO24P/lKy/U1NWHbyGzPt5+OlQzT5vZfz3eAqA4usz5q/b+Nua8j8Y1DS1Wiju/tX5e9K1YOcFrKqgK7ciGbllrZ5qQfx3MnblMc09x57EhfQcHL9254mOYSmxl25j15l0ve1Hk8pezxe7L9ZklYzKyH2MvMKKP6xJhdnhZMOGDZg2bRpmz56N+Ph49OzZE+Hh4UhOLj9NZ2dnY8yYMejXr1+lK0s1b9aWRLz6g353kiHrjiWj8cwduJ71ABtOpIge2xJ/E+/pLFZ35mYOtp9KRWGRGp9FnceLSw6LmowXRv8FANhwIgWCIGDeb+cQeykTa4+WXWsvLjmEOdvO4HZuHgLn7cHMTacBAFdvlwWqHYlp2HchAwBw8HImxq8+ju2nUvHH+eJPxNr/W7/dW9z6lPO4AIVFauQXqnHmpng6tLkTj77+/RIm/HQcn0edx9ztZ7Vev/Fp1gMW7kfgvD0mNQs/zC/EmJXHMGtLYoVdJ/NLWs3+se1sueWqS1Xe0fuHmCsYv/q4KNBqS7nzEEeuZkGtFhCffNdgK9qcbWfwqVZLYm249cTGuBS8u/ZkuUsMXEzPxdL9VzDvt3NGy5hC+1rX/js5m5qD/X/dRuylTHy797LBJQlMNWhRLF5eelgy3bzmGL3iGCb+HFfu3+mplHuYuek0budabpLA9awH6PzZH+j75X6L1cFcZs/WWbhwIcaPH48JEyYAABYtWoTdu3djyZIliIyMNLrf22+/jVGjRkGhUGDr1q2VrjDVrP+eMP7pXlfpp6zeX/xp8PET1++ixz/F/1CnrovHVK2fN528gbd6NYFfgzqif4b+EVGa7/93OlV0jJ8OX0e9Ona4n1eI9cdT8I+hbfSee/qGBOz9oA9eW158a4E/SsLK4Yi+ok9+C6P/QhtPZ0xYcwJKGznytN743u7dBBHhrbE1oez5BUHA1cwHaOLqaHS69L9/Lw5ZOJ8h2q79Pn0uNQeztyaiXys3KORyXMsqbgo+cCkTwzv5aMo9LiiCIAAOdmVjpRbvu4KYv4qnne86k44PwlpilIkDvQVBQEFRcUUuZ9xHa08no68j+2EBVHVs9bbnPi5AzF+ZCG3VEHXsyv+X8t2+y6jvaIfxPfxNqp8h1zIfwNnBFp9HXQAAbD+Vihc7eevVu+e/igeLD+/ojc3xNxHSpAHWvdUVMX/dxraEVLwb2hQ/lXTNPdvBC3HX7+KzHedRWEGAunH3IXr8cx8Wv9YJg9t56j0+Y0MCrty+j03vdIONgUUOj1+7g9i/bmNKv+awVchRUKQ2uhiiIQdKlgHQJggCHhUUac7//bxCzWNZ9/PQoK5SVHbBrgto4uqIkCau8KnvgP8cuY6CIjUm9GwiOq5Mq2NXu4Vx6LcHNN8vjP4LF2/l4vtRnUx+DYbcuPsQHip7s/dLufMQ20+lYuQzvnDVep3muJ9XiD/O30KfFm64fDsXAd4qKG3KH4+o/X8j+2EB3JwM1/1a1kNcy3qIzPv5WD42uFL103b3QT5u389DC3cnvcceFxSJxlHGXb8Ln/oOmq5J7dZqtVpA7OVM+NZ3QJOGdZ+4XlXNrHCSn5+PuLg4zJw5U7Q9LCwMhw4dMrIXsGrVKly5cgU///wzPv3008rVlJ4KN+6W3xQNAAMXxZT7eHzyPb1t2jOM2s7Zrfd4bl4hnvnsd73t0eduYdl+8eykCSXN+Hk6n8iX7b+Kuw/yNUEAKL7n0I+xSZjatxlmhLVEfqEav51KxZoj17Hs9SCT/9m+ufo40nMe6722tUeT8e3ey1jzZmd41XNAwJzdKFQL+FtPfwxs64Hgxi6irpI7D/Ixa0si5v12Fv8Y2gavdzV+Y8v/Hk/BZ1HnRd1HHw9tgzcNBIdfjiZj1pZE/H1QS7zbpxku3crFT4ev4eUgXzz3fdkNMU1Z0+aT/51Da08nHE+6iyl9m+lNbzfmXGoOBn+j34V8OeM+QiL3Ykw3PzzfwRsHL2dqAhcAbI6/CQA4fLX4TX3MymMAildfLmXspp7aHuQVwlFpgx7/LA497649iWsLhuiVK32+Y9fuoFtTV73HX15avP6Pqo4dAn1UeGnpYfx9UEtM7NUUMZduI8BbVe6bbJaBafpvrj6OfRdvI/bvofB1qQPtUV/hX8fi2Oz+KCppQcovUouu+WGBXvjtVHHgfqGjN36IuYplMVex5d1uopaT0pZMQ3ZqjTd7Uvv/uo3oc+n4aEgb5BWqsS3hJga38zR4Th7mF2pC6Be7L4p+HwVFasRdv4v3/3sKHwxsgaYN62LWlkTMCm+Nbs3Ev5f3/5uA3WfLxhaFtXHHD2PKDxLaGXbQ17H48uX2eKGjD7IfGm5FupyRW+FrN0XHT6IBAM918MLXr3RE7KXbqGOnQHzyPXy64zyWvt4JA9p44MzNbLy4pPh9+aMhrUXHiIw6j2Va/zMNXceWZlY4yczMRFFREdzdxetguLu7Iz1dv98NAC5duoSZM2ciNjYWNjamPV1eXh7y8sr+AHNyOMWTyqe7loypPjaze0O3Jal0nZpv9l7GjXuPsPnkTc1jXSP/wA+jg8o93p6z6Zj4c5zRAcylY2r6L9yPl4N9NJ/qf4xNwo+xSfjipfbI1XqTLZVXqMZHW88Ur048pA3sFHK9cn8v6QLTtuJAksFwMmtLcXfbv3ZdxLt9mmHAv4sD5M9HxN25N+4+gq9LHRSpBRSq1ShSCwY/gY76sbgF60xqNtp5q9DY1REKmQxD2uu3RPzvdCq2xt9EgpGxUUv3X9HU7Zs/LuFxgWldM2nZFQflUnO2ncFPh6/jlwldDD6eeCMbdZQKTNKebafzO1179DqauJZ9QtUemP6vXRfRsK4SH/56GioHW5yaE1Z2GEGAWgDOp+WIWiyA4sHNchmw72JxYN54IgUzwlqKypS2aC6Mvojv912BQicMlgYToPi6KX3TemHxIUzp20zzWMod4+dLLQATfjqB+c+1hVc9BwDFXXjaz3XpVi5+PXkDE3s1RX1H/fWlSk/X2JLw6OFsj9M3srHn3C1sOJ6CHVN7alqIfj+fgT4tG2L86uMG63PmZrboXE3fcApO9jbIfVyIUcuP4tqCIZqWj9v380TBBAD2nLuFOw/y4VJSzwvpOTiWdAevdfHTvCbtMThFagHTN5zCgDYeCJy/x2CdrmU9xAcbT2FyaDM0dnXUe/xWzmMUqgV4l5y/v27lQi6ToZmb4VaNbQmp+GhIG4xecUy0feLPJ1HHToFh7b0M7ncq5Z4omAD6LS5SUKlF2HQ/GQmCYPDTUlFREUaNGoV58+ahRYsWJh8/MjIS8+bNq0zViCxGO5iUeus/ceXuU9HjpfIK1XpBAAA+/FU/YGj7+UgyfjmajDZezjifVvEnN5mseHXfRwVFuJWTh80nb+h9+r+Ybvw4yXce4oeYqybPRIo+d0s0G6ZPy4FwVNrg3sN8ZD8qwPrjKVjy5xWTjgWgwmDiH7FD8705N7Es7f75l4HBjT/EXNF0MWn7du9ldGxUH0v+vIysB/misVKGlP4utVuyHhcUodU/dhnd5+3/nNDcaBQALt++j+PX7ujNsgOA7/cVn8fyxv3oDsI2Z2Xn38/fwu/nb2HRyA6YtiEBAPDnB31w7NodHE+6g41xxcE+OeshlrxeHNqjtFpcdMdL/XwkGek5xWM5zqbm4EFeIQYs3G90HSgACIn8A8M7eWteq7bcx2XhvPHM4utA5WBrdOB5p0+icfXzwQCKx8UAxR9mrnw+GAq5TBROSgUYaLXV9mvcDfwadwP/fTsEnf1dkF+oxuGrWbCVyzCqpMv57LyBkMmAsJIPAJvf7Yb3/3sKb/VqojeN29g4lof5RXrj/kqVttxpC4n8A/EfhxkobTkywYyh1vn5+ahTpw42btyIF154QbP9vffeQ0JCAvbvFw+2uXfvHurXrw+FoiyRqdVqCIIAhUKBPXv2oG/fvnrPY6jlxNfXF9nZ2XB2djbrBZqi9EIlIjLHsdn9jE7DfxJJkYMhCMDiPy/jyz3Gu1NM1bFRPYPdoRV5r19zfP3HpSd+fl3BfvWx/q2uaDZ7p2j7lL7NNIPSpWpwOw9k5uajvY8Ky3VmPppK5WCLh/mFkEGGfJ1B2I0b1NGMOavI610bGfzQoqu5W11c0ppNacgHYS0wuW9zk57XHDk5OVCpVGa/f5sVTgCgS5cuCAoKwuLFizXb2rRpg+eee05vQKxarca5c+LR4osXL8bevXvx66+/wt/fH46O+s1buir74kzFcEJEUvJsoBca1LXDqoPXLF2VavPz+C54fcVRS1eDtESEt8LbvZtW6TEr+/5tdrfOjBkzMHr0aAQHByMkJAQ//PADkpOTMXHiRABAREQEbt68iTVr1kAulyMgIEC0v5ubG+zt7fW2W1Lnxi44Vkvn2RPR02f7qdSKC9VyO89U3SBaqhqROy9UeTipLLPDyciRI5GVlYX58+cjLS0NAQEBiIqKgp9f8ayAtLS0Ctc8kZqwtu4MJ0RENaiiMThk3czu1rGE6u7Wybqfh6BP9aeZEhERWZOqnlZc2fdv3lsHEC1QRERERJbFcEJERESSwnBSoqETW0+IiIikgOGkxJPeF4KIiIiqBsNJCTsbngoiIiIp4DtyifbeKktXgYiIiMBwomHqnVGJiIioejGcaKmrrNR9EImIiKgKMZxo6dOyoaWrQEREZPUYTrTMfbatpatARERk9RhOtLhypVgiIiKLYzjR0bmxi6WrQEREZNUYTnQIkPx9EImIiJ5qDCc66thxxg4REZElMZzocHfmuBMiIiJLYjjR8WrnRgA49oSIiMhSGE50dGxUH8dm9cMvf+ti6aoQERFZJYYTA9yc7WGjkOPtXk0sXRUiIiKrw3BSjojBrZEUOdjS1SAiIrIqDCcVkMl4Q0AiIqKaxHBigi9fDoRcBqwYG2zpqhARET31GE5M8FKQD/76NBz9WrtjZLAvgOKl7t/s7m/hmhERET19uOKYiWwUxTnusxcCMDrED208nVGoFrDyYBIAYEIPfyw/kGTJKhIRET0V2HJiJhuFHAHeKsjlMtjZyPHhwJbo2dwVfx/USlPGzUmJM/MGVvo55BzmQkREVowtJ09oUmgzTAptJtrm7GCLukrzT+3FTwfBTiHH3389jY1xN6qqikRERLUKW06q0MpxwWjnrcLi1zoBAH6b3AOvdWlU7j4hTRoAKF6ZVmmjgEwmw4cDW1Z7XYmIiKSKLSdVqG8rd/Rt5a75uZ2PCu182sHD2R5KWzk+j7qgt88PY4Jw8HIW+rRsqNmmqmNr8nP2adkQf168/WQVr0YbJ4bg5aWHLV0NIiKqRdhyUgOm9GuOt3o1NfiYk70tBgV4wN5WodlmIy//1/LNqx0xrX9ztPJwwrevdqx0vRq51NF8X9qlVJFx3RrjyuemLUw3sXdTPMN7FBERkZkYTiRIIZfhxzHG11QJ8HLGtP4tsGtaLzjZ67eyfDSkNSaFNsW0/s3xyjO+cK1rZ/A42uvLKW0UCPKrX2Hd7G0VUMhlWP9W1wrLjuvWuMIyREREutitY0H9W7sZfWxAG3cMbueBqMR0vceEco45vJM3JvQU3xNoxn8TsPnkTb2yupOC7GxMz6pdS8bKGHJtwRAUFqk106+JiEj6fpvcw9JV0OC7hwU42dvgXy+2x8KRHcot9/2oTpgzrA3e6VPWJdSpUT34N3AUlWvl4aT5vmnDunrHGdre0+Dx69qLs+n859oCAJztjWdW7a4g7XrpYjAhIqpd2vmoLF0FDbac1KCGTkrczs3Dcx28MOIZ3wrLy2QyvFGyCu30/i1gZyOHIAh69/v5bUoPfLztLOKu38HoED+944S2dMP6t7rilR+OACheh+WtXk1w8HKmqJxfA0dcWzAEANB45g7RY1+/0gEX03MxIthHs+2DsJZwsFUguHF9XMt8iFlbEg2+joMz++J8ag76tXaDf0RUua952egg2NnIEdrSDX2/+hNXbz8ot7y5xvfwxwozF8uzVchQUFRee1WZqKk9cfFWDjbF3cQBnfNLRESm4cfbGvTb5B745PkAzBrc2ux9S7tcDN2I0FYhR+TwdtgzvTecDYxBkclk6NqkAYYFemFgW3ccndVPr+unIt2bFS80p90iopDLMLVfc3Rr6oqRz/ji61c6IPbvoXr7etdzQP827np1PxLRT/Tzly8HYmBbD4S2LO7u6lyFg2n3fdAHh2b2hWtdpWZb/9buRsu/FFQWwtyc7A2W+W6U/mDkNl7OeKGjD97s0djgPi6Odri2YAgih7cz+Pj7A1oAAJ7v4GW0bmSeAW3cMbyjd5Uec4iB1sg3ujeu0uewJk/z8gkce1c5DCc1yENlj9Fd/VDHzjINVt++2hHLRgdrQkJDJ6XRsl+81N6sYyvkMjzXwRu+Wt0+FfFQ2eO3yT2w+o1ncG3BEFEgAABBq7FiaHtPDAv0wq5pPQ0ea+nrQXhZZ39t/q6O8KrnINq2XOdGjv6uZd1lA9t6aL4PaWp4fE3/1u6I+6i/0ecs9UlJdxkANCl5DjcD575b0waY0q854v8xAIteKQs+fVo2xCtaLW1jtVrHOjWqV+Hzm+r1ro2wbVJ3k8tP698cXZuYHyCvLRiCawuGYPvkip8r0EeFOnYKg4+1dHcyuF1bSJMGWPJap3K7UINNGAiu66uXA9HcTdyF2trT2ezjGPLx0DZVcpza4tDMvmjnLZ3uBG3XFgzB/6Y82TiMd8vp/paSvq2Mj4G0BIYTKzYzvDX6t3bHD6OD9B57OdgXB/6vrBWkulbUb+ejQp+Whv8oQkv+WBxsFfhuVCd8+2pHtPJwhq9LWcjo2sQFFz4ZhEEBHvji5UC0cNcfc1Me7fE6bb2coSi5d0CnRvWwatwzGNzOw2hLl0wGNKirRFuv4jelb4xM6x7VxQ+b3+2GFzp6a8o0MhDi/jO+CwCgvqN4dpWbkxIfab1hDQv0womP+uOrlwNFt004PTdMtJ+3VhgzpeVABhkCfetVWK7UiGBfrH8rxOTyPZu74j/jO2t+bu9T8XO9G9pMcx3oml7SyqRrWGBxq9OQdp5Y91bXCsc/ffFyoN62hI8HlLuPva1CNNtt1RvPaK6DJ9XKwwmrxj0Dn/oOFRcG4NegDt7uXX5LqPZ1Xh3qaa3NtGB4O7g7G//go62Box3cnJTQbRDePrk7IsJb4cIng0QfoowFVV31zVgrqiIBRoLTVwauG13n5w+CbTnXX2AlxnhM6dus3Mf9Gpj+AVHbMgPvA5bEcGLFXBztsHxsMMK0Wgm0KW1M+0dQXQa2dce6v3UVhSQAeKd32R/nVyM6iNaIWTtBf4qzditQA51p1VsndcfS14PwVq8m+PT5AJyeE4bjs/ujQV0lQlu5YfFrQXBxNDIVuySybZvUHUdn9cOzgWVdMb71y/5BKOQydGpUH/8e2UHTetPc3QnLRgeJ/sHpBsCV44IxsK07Zoa3Rl2lDYa290Q7bxU6+NaDa10lXgzyEX161+3SEwQBu6b1xOTQZpj3XFuD/7C1Wx96NncFAPwyoQue6+CFjRNDDHY/Xf4sHKc+DtNriQKKux+Pze6Hib31Py0ufq0TejZvKNoWPb0XnmlcH9smdTfY/C0DMGeYfkvChU8GYWBbd4wysALzVy8H4pcJXfDVCPGbx4udfOBa1w4TeviLfqeGgne9OoZ/50DZG9/Ykvp2b9YAoS3d0NZLhbUTuuD1ro3QwMg1Y6rQVm448H99sXJcMFwc7bDqjWfQ3sgb2dxn2yIivDU2vVMWFL94qT2OzuqHy5+F48D/hSJqak84GRjoHtZGv2vzoAktGf8eKT63f8zojZeCfLDpnRC80rkR3J0Nd4Vq++bVjjgyqx9sFHK9Vtz2PvXwdu+msLdVYP+HfdC3lRu+fqUDzs0fhN9n9BK1Hq6d0AUfD20jur6Pz+6PLe92Ex0zenovrH7jGUzoYf7d5Ne8WRaqP3+hHc7MG4gXg3wwa3Aro/sMaecJBzuF6MOGblf11kndK5wl+Wrnsmu8g289vB9WfheYdnlthrrJV44LxvBO3jgzb2C5IcoSOCCWLMKUbiOZTGawS+XVzr7o0sQFjRs4alo6Sun+k7vwySBReBne0RvxyXc1U6HtbRUYFOCBQQFlAc3RxPsilX7as1HI9f4ZN3d3wpLXOsGtnE+QA9t64HLGfc3PukNudVcc/m5UJ70B0Q3qKrHzvZ5wNNBVODqkMVp5OKOVR/En+tCWbtgcXzal3Etlj93TeyEj5zEu3spFj2bF4aRbM1d0K/n+mcYuGNreExN/jsPxpLv4bUoP2CjkUNXR/0fWtYkLvh/VCQ3qKtGliQuW7r8CoDiUCAIMrsnT3N0JGycWv4k0aeiI/CI18grU2HSy+N5SvVs2RJFafGaau9XV/E4/f6EdfjmaLHrczkauqb+2r0YEas7frMGt0WRW8eBs3d+RoTdsbaXnf1TnRujgWw/NtAJi92au6N7MFTLI8J8j1/X2/W5UR6w/llI8PutSJqZtSNA85lPfATfuPkJ7rdarvq2Kuw5lMhlyHhXgvfUJouONDfHTjNEK8it782nt6ay5Jn1KgvK2Sd3R96v9ov0XvNgee85FAwCaudXFstFB8K7nUOHNR9t4qjCuW2OsPnQNQPF1+KVW0P7mlY6Yufk0jly9o7fv/6b0QH1HO1HLXuk1akgdOxusHPeM5udmbk54vasffjpcfH5Lz3lCyj1sP5UKoPhvsmOj+mjr5YyzqTkAiq+15u5OEACz7yDfq0VZqHata6e5d9pbvZri2UBvdI38Q28fuYGTaGsj3iaTyRD/jwG4lHEfgT4qTNuQgG0JqZrHz88fBAc7BdYdK77Ge5fUIzzAAzvPpGNyaDPYKGRY9PulsmMaqP9/3w5BZ38XvYkOuv9jpIThhGodmUxmcMp0qe7NGuDg5SwAEAUToPifVuRw88bT6LJTyFHX3gY2FfwHD29neAq3NgPjmysor7+D9liHv/X0x39P3MCXLwein053yJxn28Kznj2eDfTG3Yf5aF3yhuDmbA+3cj7pOtnbYu2ErhWuXbPktSDNp8Q+LRpi1RvPoIW7k+hNqDxO9rb4/IV2UKsF9G3lhg6N6kFpo8Cj/CJNmeVjgtFdJ3gYWw/IkNLzJ5fLcGxWPxSoBdSxs8GaNztj+YEkvBzkgwEl4ST+HwNw6sY9jFt1XHwMrWO19TLcwqD9a7KRy1CoFtDS3QlD23thaPviFrbnO3pj74UMzRvqvg/6oKBIrTcmrbTOz3XwRqBPPfjUd4Ci5Ji6n3b3fdAHN+8+MtgV0aRhXYS2bIh9Jbe7+GVCF7g42iHuo/5wsFOUOxZuYu+mOHwlE6duZGu2TevfHFdu38eLnfTHejV2dcT6t0JQUKRG89k7RY8Z6yYxR3N3J2yb1F0ULA39Lf17ZAf8bc0JvNevuWZbnxYN9Qtq+d+UHpjw0wlMCjU8VkT3b9BDJf7b+SCsBVYfuo6/Gxjk28LdCYPbeWL2ljOabY5KG3QoCaR9W7mJwomDTjeWUDIQb9ErHTDhZg46+NaDQi7D9oRUXM00PrOxs39xcH020EtzvUkdwwk9dUZ39cPBy1mamypWhRVjgzHx5zh88VIgBgV4QC6TGQwK5qrM3avLM3tIG8wa3Npg3VQOtvhwoPFm6IoYCyaHZvbFo4IiUfO1TCbTfKI3l1wuMzgbBgCau9fV+4f9xUuB8HB2wMqD5n0a1g5kvVo0FH06BorH/hgaG2TKr127yG9TemDZ/iuYMUD/zUr7WLYKeYVN6421Bm3bKvQr4u/qKBrYrSvAW6UJJ6WtSw3qlj8+5MD/hWpaX0o/ectkxV1fpeOkjLFVyBH791D0/Ne+cssBwJZ3u2H0imOiFpjy6I6PGtDGHdsSUkXrNLVwd8L+D8XdwuX93X7xUnsEeKtwOKKvXrnO/i44n5aDbgZac798ORCztiTixzHB6N2iISaFNhPt/9vkHtiRmIbJfZvBTiHH0at3NC2V2p4N9NJrHQOKZw9uP5WK17oWd2fprujdo7mrJpx0bGR8gPc3r3ZEQZEaO8+YFuYtieGELKK8P6AnNSjAE/s+6GPygEJT9GvtjgufhOt1Iz0pd2d7zB3WBnXsbKrs2FURmsxhaOxJVaro5TgqbTAs0NPscFJ5FZ9f7d9Ba09n0ewr845UtSaFNoONXI7+bcoPjq939cOpX08jpEkDTTDRZmj8ijHaM/gcbI2PY+vYqD7OzBto8nF1DWnnCZe/2Zk0i6tUz+aueKtXEzjYKnA06Q6Gl7QCGfob2vBWVxQUCQbHiLwU5IMXOnpr/oZ19y++CWxZi5GxwfMyWfHyDN/8cUm0/cuXAxE5vJ1J40I6+7vgpzc7Y9bmRNy890jv8QBvFcMJ1W4qh7IxAobGC1TG8dn9kXk/T9RPXx3K+/RYWVUdTEqN627+AD1rItf6R2/sza26fjeVDXqtPU17gxwW6IWtCamiGWjVyd5Wgff6N6+w3EtBPgjwVqFJQ/Hf0devdMD9vEJ4qipX34Ftq298g0wmQ7em+q0RhiwcEYgt8Tfx3audNHeBD65gXSWZTAY7G+PXQ1Vdg9P7N0d+oVrvGiovmAg6A9Z6t2gIXxcHg+FkQs/i/zd9WpbfvWVpDCdklJ2NHCdK1vEw57475WnopCx3fRUiXXY2cswd1gaPC9VGx8a09VIh2K8+PKuxFWdCD38sP5CEuc9WvA7JS0G+yH1ciC7+5Xct9m3lhh1Te8CvQdWH6Schk8kMrtvyXIcnW8xOUcEd12vK8E4+mlYSqZHJZJgZbl73q2Dgjmu6gaWU0kaBSaHlT0eWAoYTKpdrBf3RRDWhotYlhVyGX9/pVm6ZyvCp7wBHOwUc7Gwwe0hrTOnbXPNJu6L6mLIKc3mDap8mHw1pjZ+PXH+qV4K1JENBxFg4qS0YToiIjLBVyHHy4wGaAdCmBBPSN6FnE7NvmUHWTRptbEREEqW0UUhugSoibYYaSUqn3FdVl3xNY8sJERHRU2ZinyZwd1aiR3PTBglLTaUi1eLFi+Hv7w97e3sEBQUhNjbWaNnNmzdjwIABaNiwIZydnRESEoLdu3dXusJERERUpoOB+1QpbRR4pXMjg1PBawOzw8mGDRswbdo0zJ49G/Hx8ejZsyfCw8ORnJxssHxMTAwGDBiAqKgoxMXFITQ0FMOGDUN8fPwTV56IiMjavRjkg8jh7fD7jF6WrkqVkQmCeWN6u3Tpgk6dOmHJkiWaba1bt8bzzz+PyMhIk47Rtm1bjBw5Eh9//LFJ5XNycqBSqZCdnQ1n56q58ycRERFVr8q+f5vVcpKfn4+4uDiEhYlvzR4WFoZDhw6ZdAy1Wo3c3Fy4uBhf8CYvLw85OTmiLyIiIrIOZoWTzMxMFBUVwd1dvMqfu7s70tNNWw73q6++woMHDzBixAijZSIjI6FSqTRfvr6+5lSTiIiIarFKDYjVXdJZ9zbuxqxbtw5z587Fhg0b4OZm/N4OERERyM7O1nylpKRUpppERERUC5k1ldjV1RUKhUKvlSQjI0OvNUXXhg0bMH78eGzcuBH9+/cvt6xSqYRSyZVJiYiIrJFZLSd2dnYICgpCdHS0aHt0dDS6dTO+dPS6deswbtw4/PLLLxgyZEjlakpERERWwexF2GbMmIHRo0cjODgYISEh+OGHH5CcnIyJEycCKO6SuXnzJtasWQOgOJiMGTMGX3/9Nbp27appdXFwcIBK9fTfU4KIiIjMY3Y4GTlyJLKysjB//nykpaUhICAAUVFR8PPzAwCkpaWJ1jxZtmwZCgsLMWnSJEyaNEmzfezYsVi9evWTvwIiIiJ6qpi9zoklcJ0TIiKi2qdG1jkhIiIiqm4MJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpZk8ltoTSCUW8ASAREVHtUfq+be7E4FoRTnJzcwGANwAkIiKqhXJzc81aeLVWrHOiVquRmpoKJycnk24waKqcnBz4+voiJSWF66eYgOfLdDxXpuO5Mh3Plel4rkxXnedKEATk5ubCy8sLcrnpI0lqRcuJXC6Hj49PtR3f2dmZF68ZeL5Mx3NlOp4r0/FcmY7nynTVda4qc6saDoglIiIiSWE4ISIiIkmx6nCiVCoxZ84cKJVKS1elVuD5Mh3Plel4rkzHc2U6nivTSfFc1YoBsURERGQ9rLrlhIiIiKSH4YSIiIgkheGEiIiIJIXhhIiIiCTFqsPJ4sWL4e/vD3t7ewQFBSE2NtbSVaoyc+fOhUwmE315eHhoHhcEAXPnzoWXlxccHBzQp08fnD17VnSMvLw8TJkyBa6urnB0dMSzzz6LGzduiMrcvXsXo0ePhkqlgkqlwujRo3Hv3j1RmeTkZAwbNgyOjo5wdXXF1KlTkZ+fX22v3RQxMTEYNmwYvLy8IJPJsHXrVtHjUjs/iYmJ6N27NxwcHODt7Y358+ebfa+KyqroXI0bN07vWuvatauojDWcq8jISDzzzDNwcnKCm5sbnn/+eVy8eFFUhtdVGVPOF6+tYkuWLEH79u01i6SFhIRg586dmsefyutKsFLr168XbG1thR9//FE4d+6c8N577wmOjo7C9evXLV21KjFnzhyhbdu2QlpamuYrIyND8/iCBQsEJycnYdOmTUJiYqIwcuRIwdPTU8jJydGUmThxouDt7S1ER0cLJ0+eFEJDQ4XAwEChsLBQU2bQoEFCQECAcOjQIeHQoUNCQECAMHToUM3jhYWFQkBAgBAaGiqcPHlSiI6OFry8vITJkyfXzIkwIioqSpg9e7awadMmAYCwZcsW0eNSOj/Z2dmCu7u78MorrwiJiYnCpk2bBCcnJ+HLL7+svhOkpaJzNXbsWGHQoEGiay0rK0tUxhrO1cCBA4VVq1YJZ86cERISEoQhQ4YIjRo1Eu7fv68pw+uqjCnni9dWse3btws7duwQLl68KFy8eFGYNWuWYGtrK5w5c0YQhKfzurLacNK5c2dh4sSJom2tWrUSZs6caaEaVa05c+YIgYGBBh9Tq9WCh4eHsGDBAs22x48fCyqVSli6dKkgCIJw7949wdbWVli/fr2mzM2bNwW5XC7s2rVLEARBOHfunABAOHLkiKbM4cOHBQDChQsXBEEofmOTy+XCzZs3NWXWrVsnKJVKITs7u8pe75PQfcOV2vlZvHixoFKphMePH2vKREZGCl5eXoJara7CM1ExY+HkueeeM7qPtZ6rjIwMAYCwf/9+QRB4XVVE93wJAq+t8tSvX19Yvnz5U3tdWWW3Tn5+PuLi4hAWFibaHhYWhkOHDlmoVlXv0qVL8PLygr+/P1555RVcvXoVAJCUlIT09HTR61cqlejdu7fm9cfFxaGgoEBUxsvLCwEBAZoyhw8fhkqlQpcuXTRlunbtCpVKJSoTEBAALy8vTZmBAwciLy8PcXFx1ffin4DUzs/hw4fRu3dv0QJJAwcORGpqKq5du1b1J6AS/vzzT7i5uaFFixb429/+hoyMDM1j1nqusrOzAQAuLi4AeF1VRPd8leK1JVZUVIT169fjwYMHCAkJeWqvK6sMJ5mZmSgqKoK7u7tou7u7O9LT0y1Uq6rVpUsXrFmzBrt378aPP/6I9PR0dOvWDVlZWZrXWN7rT09Ph52dHerXr19uGTc3N73ndnNzE5XRfZ769evDzs5OsudaaufHUJnSn6VwDsPDw7F27Vrs3bsXX331FY4fP46+ffsiLy8PgHWeK0EQMGPGDPTo0QMBAQGi5+d1pc/Q+QJ4bWlLTExE3bp1oVQqMXHiRGzZsgVt2rR5aq+rWnFX4uoik8lEPwuCoLettgoPD9d8365dO4SEhKBp06b46aefNAPKKvP6dcsYKl+ZMlIkpfNjqC7G9q1pI0eO1HwfEBCA4OBg+Pn5YceOHRg+fLjR/Z7mczV58mScPn0aBw4c0HuM15U+Y+eL11aZli1bIiEhAffu3cOmTZswduxY7N+/v9y61ebryipbTlxdXaFQKPRSXEZGhl7ie1o4OjqiXbt2uHTpkmbWTnmv38PDA/n5+bh79265ZW7duqX3XLdv3xaV0X2eu3fvoqCgQLLnWmrnx1CZ0qZtKZ5DT09P+Pn54dKlSwCs71xNmTIF27dvx759++Dj46PZzuvKMGPnyxBrvrbs7OzQrFkzBAcHIzIyEoGBgfj666+f2uvKKsOJnZ0dgoKCEB0dLdoeHR2Nbt26WahW1SsvLw/nz5+Hp6cn/P394eHhIXr9+fn52L9/v+b1BwUFwdbWVlQmLS0NZ86c0ZQJCQlBdnY2jh07pilz9OhRZGdni8qcOXMGaWlpmjJ79uyBUqlEUFBQtb7mypLa+QkJCUFMTIxout6ePXvg5eWFxo0bV/0JeEJZWVlISUmBp6cnAOs5V4IgYPLkydi8eTP27t0Lf39/0eO8rsQqOl+GWOu1ZYggCMjLy3t6ryuTh84+ZUqnEq9YsUI4d+6cMG3aNMHR0VG4du2apatWJd5//33hzz//FK5evSocOXJEGDp0qODk5KR5fQsWLBBUKpWwefNmITExUXj11VcNTj3z8fERfv/9d+HkyZNC3759DU49a9++vXD48GHh8OHDQrt27QxOPevXr59w8uRJ4ffffxd8fHwsPpU4NzdXiI+PF+Lj4wUAwsKFC4X4+HjNVHIpnZ979+4J7u7uwquvviokJiYKmzdvFpydnWtsymd55yo3N1d4//33hUOHDglJSUnCvn37hJCQEMHb29vqztU777wjqFQq4c8//xRNfX348KGmDK+rMhWdL15bZSIiIoSYmBghKSlJOH36tDBr1ixBLpcLe/bsEQTh6byurDacCIIgfP/994Kfn59gZ2cndOrUSTSFrbYrnedua2sreHl5CcOHDxfOnj2reVytVgtz5swRPDw8BKVSKfTq1UtITEwUHePRo0fC5MmTBRcXF8HBwUEYOnSokJycLCqTlZUlvPbaa4KTk5Pg5OQkvPbaa8Ldu3dFZa5fvy4MGTJEcHBwEFxcXITJkyeLpplZwr59+wQAel9jx44VBEF65+f06dNCz549BaVSKXh4eAhz586tsemL5Z2rhw8fCmFhYULDhg0FW1tboVGjRsLYsWP1zoM1nCtD5wiAsGrVKk0ZXldlKjpfvLbKvPnmm5r3qoYNGwr9+vXTBBNBeDqvK5kg1NBygEREREQmsMoxJ0RERCRdDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCn/D7VqlAobtZFnAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(stepi, lossi);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "408cbd13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1522507667541504\n"
     ]
    }
   ],
   "source": [
    "# Validation loss\n",
    "emb = C[Xdev] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,30) @ W1 + b1) # (32, 300)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Ydev)\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "c3eb4e86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.12158203125\n"
     ]
    }
   ],
   "source": [
    "# Training loss\n",
    "emb = C[Xtr] # (32, 3, 2)\n",
    "h = torch.tanh(emb.view(-1,30) @ W1 + b1) # (32, 300)\n",
    "logits = h @ W2 + b2 # (32, 27)\n",
    "loss = F.cross_entropy(logits, Ytr)\n",
    "print(loss.item())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "471f9ae4",
   "metadata": {},
   "source": [
    "Training and validation performance are slightly, very slowly, drifting apart. **This is a sign of overfitting.**<br>\n",
    "We could now go and increase the model input from three to four characters per tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ec7747f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mona.\n",
      "mayah.\n",
      "see.\n",
      "mad.\n",
      "ryla.\n",
      "rethan.\n",
      "endrlee.\n",
      "adelynneliah.\n",
      "milopaleigh.\n",
      "van.\n",
      "aarvelynn.\n",
      "hone.\n",
      "cayshaberg.\n",
      "himiel.\n",
      "kindreellerenteromi.\n",
      "brence.\n",
      "ryyah.\n",
      "faeha.\n",
      "kayshayton.\n",
      "mahia.\n"
     ]
    }
   ],
   "source": [
    "# sample from the model\n",
    "g = torch.Generator().manual_seed(2147483647 + 10)\n",
    "\n",
    "for _ in range(20):\n",
    "    out = []\n",
    "    context = [0] * block_size # initialize with all ...\n",
    "    while True:\n",
    "        emb = C[torch.tensor([context])] # (1, block_size, d)\n",
    "        h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
    "        logits = h @ W2 + b2\n",
    "        probs = F.softmax(logits, dim=1)\n",
    "        ix = torch.multinomial(probs, num_samples=1, generator=g).item()\n",
    "        context = context[1:] + [ix]\n",
    "        out.append(ix)\n",
    "        \n",
    "        if ix == 0:\n",
    "            break\n",
    "            \n",
    "    print(''.join(itos[i] for i in out))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b11f161",
   "metadata": {},
   "source": [
    "<center>Notebook by <a href=\"https://github.com/mk2112\" target=\"_blank\">mk2112</a>.</center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}